Coding agents are among the most economically impactful deployments of LLMs, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale as models improve. Models may discover valid solutions that existing verifiers reject, and changes to the evaluation infrastructure can break embedded task assumptions. We call this accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they aim to measure.
To address this, we propose a continuous‑validation workflow that combines community reports, task artifacts, agent traces, automated oracle checks, and human‑gated repairs. We applied this workflow to Terminal‑Bench 2.0, which underwent extensive human and LM‑assisted review and red‑team testing before release.
Our audit identified issues in roughly 30% of the benchmark. After repairing these issues and releasing Terminal‑Bench 2.1, overall accuracy increased by more than 10 percentage points, leaderboard positions shifted by up to three ranks, while the overall ranking remained highly correlated (Spearman rho = 0.958).
We also examined how resource allocation shapes benchmark conclusions. A fixed wall‑clock time can materially change agent performance: extending the time limit eliminates a 20‑point gap on GPT‑5.4 and reduces a 30‑point gap on GPT‑5.4 mini to 5 points.
Thus, benchmark maintenance is not only about fixing task artifacts; resource budgets must be treated as part of the measurement specification and validated alongside the tasks themselves. As models improve, both evaluation assumptions and resource settings may need revalidation.
Review: The continuous‑validation workflow provides a systematic way to keep benchmarks aligned with evolving model capabilities, enhancing evaluation reliability and should be adopted for critical benchmarks.