On‑policy distillation (OPD) trains a student model on its own generated answers with token‑level teacher supervision. Conventional approaches assign the same weight to every token, ignoring the varying learning value of different tokens. Existing weighting schemes rely on predefined mappings from prediction signals to token weights and are not learned from the actual impact of student updates, limiting their adaptability to evolving learning needs.
MetaOPD jointly trains the student model and a lightweight token‑weighting network within a bilevel optimization framework. The inner objective updates the student using a weighted OPD loss; the outer objective optimizes the weighting network based on validation loss on reference solutions after a virtual student update. Differentiating through this virtual update links weighting decisions directly to post‑update performance, allowing the mapping from signals to weights to evolve together with the student.
Experiments were conducted on six mathematical reasoning datasets and three out‑of‑domain datasets, covering student scales of 0.6B and 1.7B and seven baseline models. MetaOPD achieved average improvements of 1.99%/5.97% (Avg@8/Pass@8) for the 0.6B student and 2.25%/6.41% for the 1.7B student over standard OPD, demonstrating robust gains across tasks and model sizes.
Review: MetaOPD leverages meta‑learning to provide a dynamic, data‑driven token weighting mechanism, overcoming the limitations of fixed mappings and offering a promising direction for enhancing on‑policy distillation performance.