Statistical privacy audits are crucial for verifying the theoretical guarantees of differential privacy (DP) mechanisms. Auditors typically employ Membership Inference Attacks (MIAs) to estimate a lower bound on the true privacy leakage at a specified confidence level, which should never exceed the DP‑provided upper bound. We discovered that many prior implementations introduce a selection bias when executing MIAs, resulting in statistically invalid and overly optimistic lower bounds. Because most audits are not tight, this bias has remained unnoticed: the estimated lower bounds stayed well below the theoretical upper bounds. Accurately evaluating this lower bound is essential, as a flawed statistical assessment can trigger false alarms for correct mechanisms or misjudge the tightness of the guarantee. This work leverages tools from the statistical literature to correct the bias, presenting an overview of several correction methods and comparing their advantages. Moreover, we conduct a comprehensive re‑analysis of previous audit results and show that without statistical correction, audits violate the theoretical upper bound up to four times more often than the claimed confidence level suggests. With correction applied, the resulting lower bounds are statistically valid and violations occur at no more than the nominal rate.
Review