Runtime failure monitors are usually judged by how well they separate failed from successful executions. Yet a perfectly accurate alarm that arrives too late cannot enable recovery, and an unnecessary intervention may disrupt an execution that would otherwise succeed.
To address this dilemma we introduce RESCUE, a counterfactual evaluation framework that measures whether a monitor’s alarms actually enable useful interventions. RESCUE restores the simulator and policy states at execution checkpoints, compares alternative actions, identifies the latest checkpoint that satisfies a predefined recovery criterion, and assesses whether acting at that point reduces the expected task‑completion cost.
We evaluate RESCUE with OpenVLA‑OFT and $\pi_{0.5}$ on the LIBERO‑Spatial and LIBERO‑Plus suites, generating 128,340 counterfactual branches. Recoverability depends on the state‑intervention pair rather than on the robot state alone: at $\tau=0.5$, only 14 of 66 OpenVLA‑OFT/LIBERO‑Plus failures contain a recoverable checkpoint under pure continuation, while the number rises to 34 when continuation or scripted recovery is allowed; in other settings, recovery actually reduces recoverability.
Under strong‑policy conditions, high detection scores do not guarantee lower intervention cost. In a stop‑and‑reset model, even an ideal detector can tolerate only about a 2.0%‑3.3% false‑alarm rate before the cost of unnecessary resets outweighs the savings from early failure stopping.
These findings demonstrate that the value of a monitor is jointly determined by the available intervention and the workload in which it is deployed.
Review: RESCUE offers a more pragmatic way to assess runtime monitors, urging researchers to consider both intervention mechanisms and task contexts when designing alarm systems.