On-policy self-distillation lets a model generate a teacher distribution conditioned on privileged information and then distill that distribution back, providing dense token‑level supervision. The privileged information changes not only what the teacher knows but also how it behaves, entangling correctness signals with unintended behavioral shifts. We compare two self‑distillation strategies on reasoning tasks: attractive self‑distillation moves the model toward a privileged teacher, while repulsive self‑distillation pushes it away. Both objectives cause strong, opposite behavior changes. Attraction suppresses exploratory reasoning, yielding shorter and more confident answers; repulsion lengthens responses, can trigger the model’s latent thinking mode, and eventually becomes unstable. Motivated by these findings we introduce contrastive self‑distillation, which simultaneously attracts the model to a teacher conditioned on a correct solution and repels it from a teacher conditioned on an incorrect solution. Unlike prior work that mixes such signals with a GRPO objective, we isolate the contrastive objective and study its behavior alone. The shared behavioral shifts of the two teachers largely cancel, leaving a token‑level signal that more directly reflects correctness. Across non‑thinking, instruction‑only, and already‑thinking models, this contrastive objective improves reasoning performance while keeping response lengths stable.
Review