This paper investigates the impact of multi‑harness reinforcement learning (RL) on coding agents. Starting from a Qwen3‑8B model warm‑started with supervised data, the authors replay frozen task‑harness records from Aider, OpenHands, Qwen Code, and SWE‑agent, applying the same number of updates under two group‑relative policy optimization (GRPO) rules: Within (one group per task‑harness pair) and Cross (harnesses pooled within a task). Each checkpoint is evaluated with a sealed SWE‑bench Verified oracle on the four source harnesses and a minimal unseen harness.
The evaluation harness dominates performance: across 24,000 sealed evaluations the mean solve rate rises from $2.14\%$ to $9.27\%$, a factor of $4.3$, whereas the training recipe only improves it by a factor of $1.16$. The grouping rule shows no significant effect. On the held‑out harness, Cross minus Within yields $+0.25$ pp (95% confidence interval $[-0.48, +1.02]$) at eight attempts per task, and $+0.16$ pp (CI $[-0.41, +0.72]$) when pooled over three training seeds, whose individual estimates even change sign. Each rule’s seed range ($0.42$–$0.45$ pp) exceeds the difference between them.
Both rules achieve their largest gains on the same source harness. An out‑of‑fold classifier can recover the generating harness from Cross’s advantage with an increase of $+4.48$ pp over a shuffled‑label baseline, while Within provides no such signal, yet both rules reach identical held‑out scores and action distributions within each harness. Re‑collecting half of the training data on‑policy does not alter these findings.
The authors conclude that cross‑harness credit yields configuration adaptation but does not enhance portable capability. They recommend that multi‑harness RL reports explicitly state the grouping boundary and evaluate under an unseen harness.
Review