LLM agents are increasingly deployed to tackle real‑world problems, and their success hinges not only on the underlying models but also on the harnesses that orchestrate their operation. Existing benchmarks mainly assess model capabilities and overall agent performance, leaving harness reliability largely unevaluated. To bridge this gap, we introduce TraceTrial‑Bench, which structures harness reliability into seven components: context, execution, governance, lifecycle, observability, tools, and verification.
We create 95 expert‑level, mechanism‑oriented tasks that provide executable or state‑based ground truth, enabling semantically equivalent comparisons while preserving each harness’s native behavior. To capture agent execution traces we design a Trajectory Tracing (TT) module and propose a two‑layer evaluation framework: the first layer measures task success, the second layer assesses harness reliability.
We also develop a Multi‑source Evidence Acquisition (MEA) module that gathers evidence from the agent’s context, the harness itself, and the execution environment for LLM‑based evaluation. Leveraging this evidence, a Failure Attribution (FA) module constructs explicit mechanism chains to attribute failures at the run level.
Across 4 harnesses, 9 models, and 95 tasks, we conduct 3,420 runs, obtaining macro‑averaged harness scores ranging from 59.8% to 72.0%. Notably, 31.1% of runs achieve task success despite violating at least one harness criterion.
TraceTrial‑Bench offers an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying the interaction between harness design and model behavior.
Review