Off-policy distillation has become central to pre-training large language models, yet the interactions between training data, objective parameterization, and model capabilities remain poorly characterized. We decompose this issue into two analyses:
- Objective-to-Capability Analysis: Investigating how training objectives shape token-level supervision and downstream performance.
- Data-to-Objective Analysis: Exploring how data heterogeneity informs objective routing.
We first show that the language-modeling objective ($L_{\mathrm{LM}}$) and the knowledge-distillation objective ($L_{\mathrm{KD}}$) induce systematically different capability profiles, tracing this divergence to a gradient-level tension between direct observed-token reinforcement and teacher-supported alternative supervision. To quantify this tension, we introduce diagnostic metrics—support coverage, observed-token probability mass, and teacher-distribution concentration—and demonstrate through controlled sweeps that the support size $k$ governs a coverage-sharpness trade-off, while distillation temperature controls within-support probability allocation.
Next, we examine adaptive objective routing: a domain-level policy applying $L_{\mathrm{LM}}$ to math and code and $L_{\mathrm{KD}}$ to general-domain data yields consistent gains over single-objective baselines, while token-level routing based on observed-token probability mass or teacher entropy fails to consistently match the single-objective baseline. These results suggest that effective objective routing depends more on the quality of the routing signal than on the granularity, reframing continued pre-training via off-policy distillation as a structured, data-conditional supervision-design problem rather than a global hyperparameter choice.
Blogger's Review: This paper delves into the significance of off-policy distillation in large language model pre-training, proposing a new analytical framework and diagnostic metrics. This methodology not only aids in understanding the formation of model capabilities but also provides new directions for future research, especially in managing data heterogeneity effectively.