ReplayRCA is a benchmark that measures whether a language‑model agent can locate the decisive mistake in a multi‑step task and fix it with a single intervention. Instead of relying on human annotations, the environment is rolled back to the step chosen by the diagnoser, the proposed action replaces the agent's original one, and the same agent continues. The metric causal repair is defined as the success rate after intervention minus the success rate of a paired placebo that replays the original action through the same mechanism.
The evaluation follows two protocols:
post‑hoc autopsy: the diagnoser sees the entire failed run and knows that it failed. live oversight: a supervisor watches the run unfold, may intervene once before an irreversible action, and does not know in advance whether the run will fail.
Tasks are drawn from OpsWorld, a deterministic procedurally generated back‑office environment with six difficulty levels; the hardest level adds multimodal elements such as QR codes and handwriting.
On 62 calibrated systematic failures, the best of 13 frontier models, Grok 4.7, achieves a causal repair of +51.3 points. The top eight models are statistically indistinguishable, the flagship models of the two largest vendors rank 8th and 12th, and five failures remain unrepaired by any model. Forcing the strongest model to intervene at the last step reduces its score by 17.1 points, showing that the last‑step shortcut used in annotation‑based attribution benchmarks does not transfer.
Across 87 runs, the two best live supervisors perform similarly, one of which is a small open‑weight model (DeepSeek V4.1 Flash). The top post‑hoc diagnoser (GPT‑6.1 Sol) is the weakest live supervisor, lagging by 14.5 points on the same runs (p = 0.002), indicating that post‑hoc diagnostic performance is a poor proxy for live oversight.
The environment and evaluation code are released at https://github.com/jeremy1392/RCAreplay.
Review ReplayRCA’s causal‑intervention framework offers a stricter test of error identification and repair, exposing a clear gap between post‑hoc diagnosis and real‑time supervision and pointing to new directions for safe AI oversight research.