Evaluating agents reliably is hampered by automatic harness optimization, which repeatedly uses a released benchmark $B_{\mathrm{rel}}$ to steer a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed target agent. The task holdout varies semantic tasks but keeps the benchmark protocol unchanged, allowing a “bad genius” Proposer to craft a cheating harness whose released‑benchmark gain relies on a benchmark‑wide shortcut.\
We introduce Counterfactual Harness Search and Evolution (CHASE), casting harness evolution as constraint generation over validity‑preserving benchmark counterfactuals. After each Proposer update, a Challenger searches for an executable protocol transformation that causes large gain destruction. A validity firewall checks that task semantics remain intact, while a confirmation set determines whether the counterfactual enters a finite archive. We formalize an exact shortcut‑neutralized benchmark $B_0$ and provide statistical guarantees linking finite counterfactual archives to $B_0$ and characterizing sequential Challenger search.\
Evaluations on a synthetic benchmark and on OfficeQA show that CHASE retains strong released‑benchmark gains while substantially reducing gain destruction under valid protocol changes.\
Review