NeFut Logo NeFut
中 Admin Login

[CS.AI] Task-Efficient Agent Evaluation Using Cached Traces

Published at: 2026-11-06 22:00 Last updated: 2026-11-07 12:32
#AI #Machine Learning #LLM

Efficient benchmarking seeks to infer a system's overall benchmark performance from only a few task executions. Conventional approaches rely solely on binary correctness of executions from already‑evaluated reference systems, overlooking the rich trace each agent produces. We first extract a structured summary of a trace using a large language model (LLM) and then embed the summary with a standard text encoder. This representation preserves correctness‑ and task‑related signals while suppressing system identity, model family, and harness‑related noise that dominate raw trace embeddings. By aggregating these embeddings across all reference systems and applying dimensionality reduction, we obtain low‑dimensional geometric representations of the systems. A new system's score is predicted from its position among the reference systems. Experiments on SWE‑bench Verified and Terminal‑Bench 2.0 show that augmenting an item‑response model with this geometry‑based prediction reduces mean absolute error from 0.13 to 0.08, with a statistically significant Wilcoxon $p$‑value.\n\nReview: This study convincingly demonstrates that LLM‑derived trace summaries can enrich feature representations for agent evaluation, enabling accurate performance estimation from a limited set of tasks.

Original Source: https://arxiv.org/abs/2611.06803

[h] Back to Home