Existing evaluation suites usually focus on a single aspect of AI agents, such as task completion (AgentBench) or security robustness (AgentDojo, ASB), while ignoring the full pipeline that includes planning, tool selection, tool execution, memory and reasoning. Failures can arise at any stage, yet traditional benchmarks rarely pinpoint the exact source. AgentAudit reads the recorded execution trace of an LLM‑based agent and assesses it across ten dimensions—instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security, and execution integrity—combined with behavioural classification and failure attribution to locate the precise stage responsible for an error. Because it attaches to the agent rather than replacing it, AgentAudit imposes no constraints on the agent’s internal implementation. We evaluated five language models (OpenAI GPT‑5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B, Gemini 2.5 Flash) on nine capability and adversarial tasks. Claude Sonnet 5 and GPT‑5 achieved the highest mean Composite Trust Scores (95.1 and 80.6 out of 100), while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash lagged behind with scores of 57.6, 45.7 and 22.6 respectively. All traces were scored by a single fixed judge model, which itself was one of the evaluated models—a limitation discussed in Section VII.E. More importantly, models with similar task‑completion behaviour can diverge sharply in trustworthiness; several non‑frontier models were repeatedly classified as Unsafe_Compliance on adversarial tasks rather than merely failing them, a nuance that pass/fail benchmarks cannot reveal.
Review