Every production model undergoes updates—retraining, fine‑tuning, quantization, or silent vendor swaps—and each update may degrade performance. We formalize this as certified paired risk‑difference auditing.
The core insight is the support identity: the risk difference between two models resides only on inputs where they disagree, which can be observed without any labels.
Leveraging this, we introduce DISCERN, a sequential two‑tier protocol. The first tier, a zero‑label tier, certifies benign updates by checking that the disagreement rate on unlabeled traffic stays below a tolerance, requiring no labels at all. The second tier, an audited tier, samples the disagreements, obtains labels, and evaluates them through an anytime‑valid confidence sequence that remains valid at any stopping time and under any label‑routing rule, even an adversarial judge.
We prove finite‑sample validity and derive matching label‑complexity bounds of order $O(\rho^2/\varepsilon^2)$. This shows that exploiting free disagreement saves a factor of $1/\rho$ compared to any pairing‑blind auditor, and the guarantee composes across an unbounded sequence of promotions within a single error budget.
Empirically, over 14,000 replayed audit streams covering 785 update pairs—including LoRA fine‑tunes of language models up to 1.4 B parameters—we observe a miscoverage of 0.0002 (nominal 5%), power of 0.986 with zero false alarms, and 56% of benign updates certify with zero labels.
Each audit emits a machine‑checkable evidence record for post‑market monitoring.
Review