In the standard GRPO (Group‑Relative Policy Optimization) setting, when every rollout for a problem fails the model receives neither a successful exemplar nor a non‑zero group‑relative advantage, causing learning to stall. Nevertheless, the failed reasoning often hints at what to try next. To exploit this, we introduce Replay‑Selected Hindsight Distillation (RSHD), which converts failed attempts into an additional, testable source of supervision.
The RSHD pipeline works as follows:
- After each failure the policy writes a corrective note based solely on its own reasoning, without seeing a reference answer.
- The note is fed back as context and the problem is solved again, producing a new rollout.
- If the new rollouts succeed more often than the original group and the note does not simply state the answer, the note is admitted.
- Admitted rollouts are distilled into the plain‑prompt policy for further training.
Admission is decided by fresh successes rather than the original failure, allowing supervision to be recovered even from groups where every attempt failed, and it works equally well for mixed‑success groups. The only component trained is the policy; the sole judge is the outcome verifier, and deployment requires no notes or extra inference.
Across several model families, sizes, and two training domains, RSHD consistently outperforms vanilla GRPO on in‑domain benchmarks, and the gains transfer to out‑of‑domain evaluation sets.
Review