Synthetic data can break the bottleneck of human‑generated samples, yet it may also cause model collapse: even a fixed fraction of synthetic points imposes a non‑vanishing excess risk floor. This work studies one‑pass SGD for high‑dimensional linear regression ($d\gg n$) under two curricula – mixed training and two‑stage training.
We first derive finite‑sample risk bounds that separate the classic bias‑variance decomposition from source‑mismatch effects. For mixed training the risk can be expressed as $$R_{mix}=\underbrace{\text{bias}^2+\text{variance}}_{\text{standard}}+\underbrace{\text{fluctuation}+\text{persistent\ drift}}_{\text{synthetic induced}}.$$ Fluctuation captures random mixing noise, while persistent drift accumulates the distribution shift between real and synthetic data. For two‑stage training the bound reads $$R_{2stage}=\text{bias}^2+\text{variance}+\underbrace{\text{filtered\ initialization\ bias}}_{\text{appears only in stage 1}}.$$
The bounds reveal a sharp contrast: mixed training inevitably leads to strong model collapse, leaving a risk floor that does not shrink with more real samples; two‑stage training uses synthetic data only in the first phase and then fine‑tunes on real data, thereby eliminating the floor.
Under a random sketch model we obtain scaling laws. In the optimization‑saturated regime (SGD has converged), mixed training’s risk scales roughly as $R_{mix}\sim \frac{m}{n_{real}}\cdot\epsilon_{syn}$, indicating that larger models amplify synthetic‑induced degradation. Conversely, the bias term of two‑stage training decays with synthetic pre‑training quality $q$ as $\text{bias}\sim \frac{1}{q}\cdot\frac{1}{\sqrt{n_{real}}}$.
We also provide an exact finite‑sample necessary‑and‑sufficient condition under which two‑stage training strictly outperforms real‑only training given the same real‑data budget and identical real‑stage updates.
The overall message is that synthetic data is neither inherently harmful nor beneficial; its impact hinges on data quality and how it is incorporated into the training pipeline.
Review