With the rapid proliferation of large language model agents, understanding and diagnosing their failures is essential for improving system effectiveness and trustworthiness. Agent failures often appear as long and complex trajectories, making manual frame‑by‑frame inspection infeasible.
Traditional software bug diagnosis techniques do not translate well to LLM agents, and relying solely on LLMs for judgment lacks reliability.
To address this, we introduce AGENTSCOPE, a neuro‑symbolic diagnosis framework. Its key idea is to abstract agent behavior from trajectories into structured representations and to define neural invariants that capture behavioral properties.
On top of the structured representation, AGENTSCOPE employs LLM‑guided reasoning against the neural invariants to pinpoint the failing step and its type within the trajectory.
We evaluate on the public Who&When dataset and a newly created AgentErrata dataset, where AGENTSCOPE markedly outperforms current state‑of‑the‑art methods in fault localization and attribution accuracy.
These results demonstrate that combining structured abstractions with LLM reasoning enables efficient, reliable, and interpretable diagnosis of agent failures.
Review