On-policy distillation (OPD) supplies teacher supervision on trajectories generated by the student, yet each trajectory can realize only one continuation at every decision, limiting supervision coverage under a finite sampling budget. We introduce BranchOPD (Branching On-Policy Distillation), which selects informative decision points based on student uncertainty and teacher‑student disagreement, resamples student‑policy continuations from shared prefixes, and applies token‑level teacher supervision to both main trajectories and branches. BranchOPD operates at token‑level decisions in mathematics and within‑turn decisions in agent tasks. Reusing shared prefixes reduces student‑generated token costs by about 19% on ALFWorld and 31% on WebShop compared with seven independent trajectories. We also propose protocol‑boundary repair (PBR), which delivers targeted supervision under protocol constraints to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi‑Rollout OPD (n=7) across all evaluated benchmarks, achieving a 18.55‑point gain in mean mathematics accuracy over 16 sampled answers (avg@16) and a 13.8‑point gain in mean ALFWorld ID/OOD success. Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and surpasses random selection and turn‑initial branching.
Review