Large language model (LLM) agents deployed in production remain unreliable on long, multi‑step workflows, even as benchmark success rates steadily improve. We argue that this gap is largely due to task horizon: public benchmarks are dominated by short‑to‑medium horizons where success stays high, while real‑world workloads require an order of magnitude more dependent steps.
We measured this effect directly through a large controlled study covering nine models (including six open‑source models ranging from 1.2B to 671B parameters) and three proprietary systems, four task families (one of which is a genuine tool‑use loop), five horizon lengths, and three context regimes. Task success follows a geometric decay law governed by a single per‑step reliability parameter $r$. Although $r$ increases with model scale, it saturates well below 1 even for the strongest models, guaranteeing eventual collapse at sufficiently long horizons.
The degradation is most pronounced on the agentic task: every model tested, including widely deployed systems, drops from near‑perfect success to near‑zero within sixteen steps (analyzing 10,664 trajectories). Further analysis shows that decay is driven by step count rather than context length; limiting the context window actually steepens the decay (logit slope -0.69 vs. -0.44, $p=3\times10^{-6}$), contradicting the “lost‑in‑the‑middle” hypothesis and warning against a common production shortcut.
Projecting the measured reliability onto representative benchmark horizons quantifies a substantial gap between benchmark and production conditions: reliability is about 0.42 at GAIA‑length horizons but falls to roughly 0.24 at hundred‑step production horizons. For teams responsible for large‑scale agent orchestration and reliability, these findings argue for horizon‑aware evaluation and reliability budgeting instead of aggregate pass‑rate metrics. Code, prompts, seeds, and raw trajectories have been released.
Review