This study audits various code optimization benchmarks, such as GSO, SWE-Perf, and SWE-fficiency, which evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Although leaderboard scores are increasingly used as evidence of coding-agent progress, they can conflate runtime instability, benchmark-specific scoring rules, and the number of tasks already solved by at least one public submission.
First, we replayed the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks are replayable, but their reference patches satisfy the original benchmark validity rules in only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks during cross-machine replay; SWE-Perf is particularly fragile as many reference patches yield near-zero runtime changes.
Second, we show that public submission rankings strongly depend on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns excessively high score weights to the worst ten tasks, ranging from 58.5% to 82.8%.
Third, by examining 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks and surpasses the unoptimized base code on 99.8% (449/450).
Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing remaining performance gaps concealed by aggregate rankings.
Blogger's Review: This analysis highlights critical concerns regarding benchmark reliability and scoring methods. The findings suggest that while benchmarks serve as indicators of progress, they must be treated with caution due to inherent inconsistencies and performance variances across different environments.