Tool‑using agents fail in two ways: picking the wrong tool or constructing wrong arguments. An early mistake of either kind can silently corrupt downstream steps, causing the whole task to collapse. We measured a correct‑invocation rate on five open‑weight models across contamination‑free multi‑step tasks (depths 1‑8), both under a clean teacher‑forced context and the model’s own free‑running context. By depth 6, roughly 70% of a model’s clean‑context capability is lost to its own earlier errors ($L_6 = 0.686$, $L_6 = 0.684$).
The central finding concerns the measurement itself. When scoring against a fixed gold trajectory with exact‑match criteria, a propagation model’s severity and recovery parameters are not merely hard to estimate—they are fixed by the scoring rule. Severity is forced to its boundary (0 of 869 poisoned steps correct); recovery is structurally unobservable (0 of 580 poisoned steps returned on‑track, against an expected 0.0058 by chance). Both stem from one mechanism: after divergence, the gold value is generated by tool constants the model never sees, so the information is inaccessible to the model. Yet a fit still returns 0.92 and 0.73 for a quantity that is exactly 1.000—confident numbers for a parameter already determined by the scoring rule.
We propose a remedy—conditional‑on‑state scoring—which can be applied retrospectively to cached completions at zero additional cost, un‑pinning severity from the zero boundary and yielding interior estimates (+0.149, +0.316).
Blogger's Review: The paper exposes a hidden bias in exact‑match evaluation for multi‑step tool‑using tasks and offers a practical, cost‑free post‑processing fix, making it a valuable contribution to reliable agent assessment.