Recursive self‑improvement (RSI) has attracted attention for its ability to learn with limited data. In a proposer‑solver loop, a proposer model creates training tasks while a solver model learns to solve them via reinforcement learning; this loop has succeeded in mathematics and coding because answers can be automatically verified. Open‑ended expert tasks such as clinical report generation lack an answer key, making simple Q‑A generation infeasible. The proposer must therefore also write a rubric that lists required content and prohibited errors. Naïvely generated rubrics are often incomplete, allowing solvers to game them by naming expected topics without providing substance, which corrupts learning. To address this, we introduce Recursive Reward Improvement (RRI), which employs a reward‑hacking adversary to supervise the reward. The procedure works as follows: before the solver trains on a generated task, an adversary writes an answer that exploits the rubric while being as useless as possible; the same model then writes an honest answer without seeing the rubric; up to two repair steps revise the rubric so that the honest answer wins. Discovered exploits are fed back to improve future task and rubric generation. Experiments on HealthBench Professional show that a 9B model improves the adjusted score by 10.9 points over the generation‑only baseline, a 27B model by 12.4 points, and gains 10.3 points without any external model. Gains are largest on difficult and writing tasks. Moreover, the 27B model trained on HealthBench Professional descriptions improves by 6.6 points on PRBench Hard and 8.0 points on ProfBench without seeing any tasks from those benchmarks. We also analyze exploitable points in model‑generated question‑rubric pairs to aid future research on open‑ended self‑play.
Review