Abstract
Automated failure attribution uses large language models (LLMs) to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems.
Benchmark Construction
Using a strictly controlled pipeline, we inject failures only after exactly replaying a successful prefix, constructing 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios.
Experiments and Analysis
Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, providing empirical guidance for future automated failure attribution systems.
Blogger's Review: The introduction of Who&When Pro marks a significant breakthrough in the field of AI agent failure attribution. Through systematic experimental design and dataset construction, it lays a crucial foundation for future research, enhancing the accuracy of failure identification and reinforcing the safety and reliability of AI systems.