Data‑science and ML‑engineering agents are typically evaluated on benchmarks such as MLE‑bench, ALE‑Bench, SWE‑efficiency and AlgoTune, where each task yields a scalar metric (Kaggle score, contest ranking or speed‑up). When the gap between top models vanishes, the usual fix is to introduce a harder version. By rescoring eight benchmarks—including SWE‑bench Verified and Terminal‑Bench—we discover that most of the apparent saturation stems from the measurement methodology rather than the tasks themselves.
Most agent benchmarks collapse a run into a single pass/fail bit, which is problematic in two ways:
- Too coarse: on SWE‑bench Verified, the top‑ten agents differ by less than the test’s error margin; correcting the errors merely reshuffles neighbours, and deleting a random sixth of each agent’s passes can reorder the ranking just as much.
- Capped by the reference: ALE‑Bench’s top ten tie when graded against the median human, even though nearly half of the runs outperform that human.
On fixed tasks, raising the bar only shifts the problem; graded credit resolves it. We introduce a clipped log‑ratio score with dated anchors and treat cost as a separate axis. In a Terminal‑Bench re‑run, models tied on pass/fail differ by more than an order of magnitude on the quantitative metric that the tests discard.
Review: Finer‑grained measurement and multi‑dimensional scoring are essential to break through the current ceiling of agent benchmarks.