Self‑distillation can improve reasoning models when no external teacher is available. Existing approaches usually rely on privileged information to build a stronger self‑teacher, which limits their use with only unlabeled prompts. We introduce Weakening‑Induced On‑Policy Self‑Distillation (WISD), which first creates a weakened counterpart of the original model and then uses the contrast between them as a training signal, requiring no privileged data. Two weakening strategies are investigated: low‑bit quantization and magnitude pruning. Across diverse architectures and scales, we evaluate WISD on mathematical and coding reasoning tasks. Quantization raises average accuracy by about 4.0 pp, pruning by about 5.5 pp, achieving state‑of‑the‑art results on all benchmarks. Notably, pruning lifts Qwen3.5‑4B’s accuracy from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization lifts Qwen3.6‑35B‑A3B’s accuracy from 68.5% to 74.6% and from 61.8% to 69.5% on the same tests. These findings establish WISD as a novel and effective self‑distillation framework for large reasoning models.
Review