On‑policy distillation (OPD) speeds up post‑training by giving dense token‑level supervision from a frozen teacher on the student’s own rollouts. Vanilla OPD applies this supervision uniformly across prompts without checking whether the teacher is reliable for each prompt. Because reverse KL ($\text{KL}(p\|q)$) is mode‑seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies such as entropy or teacher‑student likelihood agreement only measure uncertainty or agreement and do not directly verify correctness.
We introduce Teacher‑Gated On‑Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier‑scored teacher probes; each prompt is routed either to dense OPD if the reliability check passes, or to verifier‑grounded GRPO otherwise.
Across 4B and 35B student models in mathematics, code, and instruction‑following tasks, TGOPD outperforms vanilla OPD in all six single‑domain settings and achieves higher averages on seven benchmarks under multi‑domain training.
By using otherwise idle teacher capacity for reliability estimation, TGOPD also reduces teacher‑side compute waste in asynchronous OPD, raising teacher‑node GPU utilization from 9.8% to 78.9% in the measured 4B single‑domain run.
Review