NeFut Logo NeFut
中 Admin Login

[CS.AI] Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Error Entry and Control Reach

Published at: 2026-11-11 22:00 Last updated: 2026-11-12 00:51
#AI #Machine Learning #LLM

This paper conducts a fine‑grained analysis of failures in long‑horizon LLM agents. Conventional studies only log symptoms—such as a failure‑mode label or the step deemed responsible—without indicating when the error entered, how it propagated through states, whether the trajectory already contained contradictory evidence, or which control mechanisms could have intercepted it. To address this gap we introduce an episode record that ties a candidate fault to its origin, propagation, detection, response and consequence, and annotates the control families that observe and reach the fault in a given operational phase and state domain. We turned the record into a written annotation protocol and applied it to every trajectory of 95 failed runs from the public Who&When and MAD corpora (six multi‑agent systems). Faults tend to appear early (median at 0.18 of the run), mainly as wrong decisions or fabrications about task state; in 37 of 88 Who&When runs the fault occurs before the step the corpus human labels mark as decisive. In 72% of failed runs the trajectory already contained evidence against the error, yet in 88% of those cases no agent acted on it. Deviations were flagged in 31% of runs, but no subsequent response corrected the deviated state. Independent judges reading the unused evidence, task contracts fixed at admission, and re‑checking beliefs against the environment reach most episodes; simply restoring the agent’s own state would have sufficed in 35 of 95 runs. A raw‑corpus text scan also shows that 45% of hand‑crafted Who&When failures end with a provider content‑filter crash after which the harness emits a final answer that no agent produced, a failure class absent from symptom taxonomies or responsible‑agent labels. The protocol, records and scripts are released with the paper.\n\nReview: The study offers a systematic fault‑tracing framework that makes error origins, propagation paths and missed control opportunities explicit, paving the way for more dependable long‑horizon LLM agents.

Original Source: https://arxiv.org/abs/2611.13254

[h] Back to Home