Abstract
Large language models (LLMs) have made automated program repair (APR) increasingly practical for real-world bugs, but repairing directly from bug reports remains underconstrained. Bug reproduction tests (BRTs) help close this gap by turning a bug report into an executable, bug-specific signal that can guide repair and validate candidate patches. Existing work has therefore studied BRT generation as a core subproblem in APR and mainly evaluates a generated BRT using the fail-to-pass (F-P) criterion, which requires the test to fail on the buggy code but pass on the golden fix.
We show that F-P alone is insufficient when the goal of a BRT is to improve downstream repair. In particular, some F-P BRTs are lax, reproducing the observed symptom yet still admitting plausible-but-incorrect patches. We formalize this missing quality dimension by separating F-P BRTs into rigorous and lax ones, and show empirically that only the former consistently improve repair success.
We further find that co-generation introduces test-fix error coupling, where the in-trajectory fail-to-pass (F-P) check can pass even when both the generated patch and generated test are wrong. Based on these findings, we propose CoHarden, a co-generation framework that uses the Lax signal as an in-loop convergence criterion. CoHarden first generates a test before any fix, then iteratively hardens the test and fix against surviving mutation patches until the generated test no longer admits Lax regressions.
Experiments show that CoHarden reaches 69.4% Resolved and 78.9% F-P on SWE-bench Verified, outperforming the strongest fix-only and co-generation baselines by +9.6 and +7.9 percentage points in Resolved, respectively, with consistent gains across LLM backbones and benchmarks.
Blogger's Review: The CoHarden framework effectively addresses the shortcomings of the traditional F-P criteria by introducing an iterative hardening mechanism, showcasing innovative potential in the field of automated program repair. This not only enhances repair success rates but also provides new directions for future research, making it worthy of attention and further exploration.