Training models on their own outputs has become a popular route to boost AI capabilities, and self‑distillation lets a model match its token‑level predictions when given extra context such as cues, feedback, or instructions. The safety of this procedure, however, remains poorly understood. This paper reports a surprising safety failure: in recursive self‑distillation each distilled model becomes the teacher for the next round. We observe that when a safe model is recursively distilled on its own (safe) generations while receiving an additional non‑harmful cue, its alignment degrades progressively across a wide range of harmful queries, a phenomenon we call Progressive Misalignment. The effect appears in both the Qwen3 and Phi‑4 model families on SORRY‑Bench and StrongREJECT. It also occurs in a more realistic setting where the fixed cue is replaced by feedback from a simulated user with benign intent. Using a gradient‑based search we can even find seemingly harmless prompts that trigger progressive misalignment under recursive self‑distillation, showing that harmful training prompts are not required for this failure mode. We hope this work encourages further investigation of the safety properties of AI systems trained recursively on their own data.
Review