On‑policy distillation (OPD) offers dense token‑level supervision for language model fine‑tuning, yet its performance hinges on teacher quality. External teachers suffer from distribution mismatch, while self‑distillation with privileged conditioning is limited by the model’s in‑context learning capacity.
RISE (Recursive Improvement via Self‑Extrapolating Policy Distillation) builds a synthetic teacher directly from the model’s own RLVR training trajectory. It extrapolates the displacement between the current checkpoint and a trailing anchor—either in parameter space or output logit space—turning a sparse outcome‑driven parameter update into a dense token‑wise target, without any external model or privileged conditioning.
RISE couples RLVR and OPD in a complementary loop: outcome rewards guide the extrapolation toward correct reasoning, while the extrapolated teacher refines token‑level decisions. Because the teacher is refreshed each iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one‑shot compression step.
Experiments across mathematical reasoning, multi‑domain STEM, code generation, and multi‑turn agentic tasks show that RISE consistently outperforms RLVR‑only training and on‑policy self‑distillation.
Review: RISE demonstrates how a self‑generated teacher can provide dense supervision while remaining anchored to result‑based feedback, highlighting the promise of recursive improvement for large‑model fine‑tuning.