Abstract
Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting. A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark. We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench.
Findings
A partial budget counts as enough only when it supports the completed benchmark's decision, covers required task groups, and leaves no more than a target fraction of comparisons unresolved. The required task fraction varies sharply. At the strict 0 percentage point threshold on a 5 percentage point budget grid, AppWorld first meets all targets at 15 percent, tau-bench at 25 percent, and SWE-bench Verified at 90 percent; SWE-bench Lite does not meet all targets by 95 percent under the primary coverage rule.
Evaluation Report Recommendations
Partial-evaluation reports should state how much one agent must outperform another, how tasks are selected, what coverage rule is required, what decision rule is used, and how many comparisons may remain unresolved.
Blogger's Review: This article explores the effectiveness of agent benchmarks through replay analysis, emphasizing how to ensure the reliability of evaluation results in partial task execution scenarios. The findings provide critical guidance for assessing agent performance, particularly under resource constraints. It is advisable to clarify task selection and decision rules when conducting partial evaluations to enhance result validity and transparency.