Classification accuracy and true‑positive/false‑positive rates alone do not determine whether restarting an AI agent improves overall success. By leveraging an exact alarm‑to‑action identity for an independent clean restart with zero monitoring overhead, we analytically show that a monitor fixed at 90% accuracy can swing the success rate anywhere from ‑5% to +35%, solely depending on the residual‑budget success probability $p_{\text{budget}}$ of the replacement attempt.
We introduce two decision‑oriented evaluation techniques. The first is a decision‑sufficiency criterion, a binary‑action specialization of Blackwell’s comparison of experiments, which identifies whether a given progress representation or context compression preserves the information needed to optimally choose between the actions “continue” and “restart”, independent of trace‑reconstruction fidelity.
The second is a finite‑sample, model‑free decision audit. This method builds simultaneous binomial confidence intervals over existing agent traces without additional model calls, allowing us to certify a preference for “restart” or “continue” under stated sampling assumptions, or to return “abstain” when evidence is insufficient.
In a study of 194 externally evaluated model executions on 19 selected SymPy issues from SWE‑bench Verified (with a retrospective audit of 130 traces), we observed:
- An apparent restart gain of +2.29% traced to a single surviving failed attempt;
- A loss of ‑42.29% on a second model;
- No positive replication of the initial gain on prospective task cohorts;
- Zero out of 103 live comparisons meeting the certification bar.
These findings motivate evaluating agent monitors through recovery opportunity, decision value, and statistical support, rather than classification performance alone.
Review