In clinical practice, agreement among independent experts is taken as evidence of reliability, and multi‑round consensus has become a core mechanism for medical question‑answering systems. When such a system must decide whether to trust its own answer, the prevailing signal is still the agreement among sampled answers. However, agreement is a fragile proxy for correctness. A system can be unanimously wrong, returning the same incorrect answer on every sample, and agreement‑based signals then carry no information. The root cause is that these signals only observe the final consensus state and discard how it was reached. Agreement achieved by resolving disagreement with evidence looks identical, at the end, to agreement that stems from a shared misconception from the first sample. ProbeGuard is a certified abstention framework that bases the abstention decision on the formation process of the consensus. It tracks agreement trajectories, minority persistence, and retrieval saturation. For unanimous votes, rationale semantic entropy checks whether the reasons behind the vote cohere, and an active probe retrieves counter‑evidence to test whether the consensus survives. A stratified Learn‑then‑Test calibration then converts these scores into a distribution‑free bound on selective risk. We evaluate ProbeGuard on three medical QA benchmarks and a hard‑frontier reference, using a published multi‑round RAG substrate, against six abstention baselines. On MedQA, 13.4% of unanimous votes are wrong, and no agreement‑based signal can flag them. Process signals raise the discrimination of correct versus incorrect consensus from chance to 0.696 AUROC. The certified rule answers six in ten unanimous‑layer questions at an observed selective risk of 9.0%, and nine in ten once in‑domain calibration data accumulate.
Review