Data mixing is a central design problem in large language model (LLM) pretraining: given a fixed token budget, practitioners must decide how much data to allocate to each domain. Recent proxy‑based methods address this by training small models on candidate mixtures, fitting a response model, and using the response to select mixtures for large‑scale training. We show that this workflow is essentially a classical mixture experiment.
Under this view, data domains are mixture components, token shares are component proportions, proxy‑training runs are experimental design points, and validation loss defines a response surface over the probability simplex. We develop a sparse second‑order $\text{Scheffé}$ response‑surface model and construct model‑robust $\mathcal{I}$-optimal designs for proxy data‑mixing experiments.
Using RegMix as an empirical case study, we demonstrate how the framework can both interpret observed mixture responses and design more efficient proxy experiments. The $\text{Scheffé}$ analysis reveals that domain value is strongly relational: several domains that are weak under additive effects become favorable through pairwise interactions, especially when combined with web‑derived text. The sparse $\text{Scheffé}$ model preserves mixture rankings across model scales and remains competitive with a flexible machine‑learning predictor while providing an explicit decomposition of additive and interaction effects.
In a simulation study calibrated to observed proxy‑training responses, model‑robust $\mathcal{I}$-optimal designs recover the relevant mixture ordering after removing about 25% of the original proxy runs.
These results suggest that LLM data mixing should be treated not only as a prediction problem but also as an experimental‑design problem, where the proxy mixtures themselves can be chosen to improve statistical efficiency.
Blogger's Review: The paper elevates LLM data mixing to the realm of experimental design, offering an interpretable and efficient framework that can substantially guide practical resource allocation in pretraining.