NeFut Logo NeFut
Admin Login

[CS.AI] SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

Published at: 2026-08-07 22:00 Last updated: 2026-08-08 01:08
#LLM #Open Source #SearchAuditor

The SearchAuditor is a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors. Blogger's Review: The SearchAuditor framework provides an effective solution for addressing the failure issues in search agents, and its multi-perspective auditing method and repair capability make it have a wide range of applications in the field of search agents.

Original Source: https://arxiv.org/abs/2608.05212

[h] Back to Home