Harness evolution automatically optimizes prompts, skills, tools, and orchestration code around a fixed‑weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively refined on a search set to achieve higher performance, but it is unclear how much of the observed gain stems from benchmark‑specific overfitting, extra inference‑time compute, or truly generalizable improvement.
This paper introduces Harness‑Delta Attribution (HDA), an evaluation protocol that decomposes an evolution gain into three components:
- Artifact‑driven overfitting – learning patterns that only benefit the search set;
- Test‑time scaling – gains attributable to increased inference computation;
- Residual generalizable improvement – an upper bound on the remaining improvement that may transfer beyond the search set.
We apply HDA to 11 executor models and three harness‑evolution frameworks across math, coding, creativity, and agentic tasks, covering 21 settings with statistically significant search‑set gains. On average, 54% of the gain is explained by overfitting and 14% by test‑time scaling; together they exceed 50% in 14 settings. The residual and held‑out transfer vary widely across tasks, models, and frameworks. Importantly, gains on the search set often fail to transfer to unseen tasks and provide limited evidence of persistence under realistic distribution shifts. Validation‑based signals are insufficient to prevent overfitting, highlighting the need for attribution‑aware evaluation in future harness evolution research.
Review: HDA offers a fine‑grained lens for assessing automated agent improvements, urging researchers to look beyond raw search‑set scores and focus on genuine, transferable progress.