NeFut Logo NeFut
Admin Login

[CS.AI] Extremely Sparse Supervision Incentivizes Reasoning Ability

Published at: 2026-09-07 22:00 Last updated: 2026-09-08 00:37
#Machine Learning #LLM #Artificial Intelligence

Large language models exhibit increasingly strong reasoning abilities after post‑training. Conventional post‑training optimizes over massive numbers of tokens, implicitly assuming that effective learning must be token‑intensive. We revisit this assumption within the on‑policy distillation (OPD) framework, which naturally provides dense teacher supervision at every generated token. Using the Qwen3 family, we unexpectedly discover that marking only a tiny fraction of tokens—one to two per reasoning trajectory, about 0.05% of all tokens—sufficiently boosts reasoning performance. Sparse supervision matches or even surpasses full‑token training in most cases, despite excluding the vast majority of generated tokens from the training objective. This phenomenon consistently appears across nine teacher‑student configurations of varying scales, covering mathematical reasoning, coding reasoning, Llama models, and PPO‑based reinforcement learning with verifiable reward (RLVR). We argue that such extreme sparsity mirrors natural learning: rather than correcting every step word by word, one reflects on a few critical steps, updates understanding, and proceeds, avoiding micro‑level corrections while remaining highly effective. Overall, the results challenge the belief that post‑training must be token‑intensive and point to a new direction for designing more efficient post‑training algorithms. Review

Original Source: https://arxiv.org/abs/2609.04565

[h] Back to Home