Abstract
In LLM-based agentic systems, failure attribution, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale.
We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space.
At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200–5000 times faster than prompting-based baselines, and consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively. This demonstrates that OAT is a promising and efficient direction for diagnosing agentic system failures.
Blogger's Review: The OAT method showcases significant potential in unsupervised learning, especially in resource-constrained environments. By leveraging the dynamic patterns of successful trajectories to infer failure steps, it not only saves computational resources but also provides a fresh perspective for failure diagnosis, warranting further exploration and validation in more application scenarios.