When solving multi-step tasks, large language model (LLM) agents often commit to trajectories that are doomed to fail, yet they consume significant inference compute before the failure becomes apparent.
We demonstrate that failure can be predicted early from the agent's internal representations: lightweight per-round probes on hidden activations can anticipate eventual episode failure as early as the first interaction round, while scorers relying solely on observable behavior are barely better than chance.
We translate this signal into a practical abort cascade: one distribution-free calibrated gate per round, with per-round recall budgets jointly searched to ensure that eventually successful episodes survive all gates at a user-specified global rate. This episode-level guarantee is crucial in deployment, as false-abort risk accumulates across gates.
Across two agent models on TextCraft, the cascade meets every recall target from 90% to 97%, and at the 90% target, it saves 47.1% +/- 10.3% (Qwen-2.5-7B) and 37.2% +/- 8.8% (Llama-3.2-3B) of inference compute, achieving 1.6–1.7x the best single-gate policy.
An otherwise identical cascade reading only behavior saves roughly half as much, and adding behavioral features to the probe yields no further gain: the hidden states capture what behavior reveals.
Finally, we characterize the sample complexity of certifying high recall targets, informing practitioners which recall promises their data can and provably cannot back. The code will be released soon.
Blogger's Review: The proposed recall-controlled probe cascade method effectively predicts LLM agent failures, significantly enhancing computational resource efficiency. Its potential applications should not be underestimated, especially in terms of cost savings and performance improvement. The key lies in balancing recall rates with computational efficiency, providing a crucial reference for future LLM applications.