In reinforcement learning, sparse verifiable rewards often fail to give fine‑grained learning signals. On‑policy self‑distillation addresses this by having a teacher, conditioned on a reference solution, score the student’s full rollout and distill the resulting distributional shift back into the policy, aiming for dense token‑level credit. We first examine whether this scoring signal actually contains corrective information. Using gold error spans annotated in student rollouts, we evaluate the two signals used by self‑distillation objectives—the full‑distribution $KL$ and the token log‑ratio—on two criteria: error localization and the ability to push probability away from erroneous tokens. Experiments show that neither signal reliably localizes errors nor distinguishes correct from incorrect rollouts of the same problem, indicating a lack of corrective power.
We argue that this failure is structural: locating an error in a reasoning trace is itself a search problem, while a scoring teacher is never explicitly asked to perform that search. The same model, given the same privileged information, localizes errors much better when it is first asked to generate a critique of the current rollout. This motivates critique‑conditioned self‑distillation: first generate a critique, then condition the scoring teacher on that critique. The resulting signal localizes errors, provides corrective feedback within error spans, and shifts probability toward tokens that trigger backtracking such as “Wait”, “But”, and “Actually”.
Further evaluation shows that the critique is useful at test time: resuming a wrong rollout with the critique in context turns math problems that the model could not solve by sampling into solvable ones. Moreover, in cold‑start GRPO, the critique supplies an advantage signal for prompts where verifiable rewards alone give none.
Review