On‑policy distillation (OPD) combines student‑generated rollouts with dense token‑level supervision from a teacher. Prior work mainly examined algorithmic behavior, leaving the role of training data unclear. This study pushes the data‑minimal limit by training on a single query and investigates the outcome.
Results show that one‑shot OPD keeps improving for hundreds of steps and recovers most of the gain of full‑data OPD across task domains and model families. We explain this via state coverage, the fraction of states visited by full‑data OPD that are reached by the query set’s rollouts. A single query already attains about $71.5\%$ coverage, most of it within the first 100 steps.
Adding semantically distinct queries raises both coverage and validation accuracy; with 16 queries coverage reaches $98.9\%$, matching full‑data training. However, the alignment rate between student and teacher slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states requires hundreds of steps to be fully absorbed. Hence OPD is data‑overfed but algorithm‑starved: rollouts quickly expose broad supervision, while the student absorbs it increasingly slowly.
The state‑coverage finding extends to multi‑teacher OPD (MOPD); 16 semantically diverse queries per domain match full‑data MOPD performance. As a stress test, content‑light templates and off‑domain WildChat queries also approach the real‑query baseline, indicating that task content and induced state coverage can decouple.
Review: These insights suggest future work should focus on step‑efficiency of OPD and re‑examine the interplay between data scale and underlying mechanisms, enabling efficient distillation of large models with minimal high‑quality queries.