Abstract
Reinforcement learning (RL) with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self-generated rollouts. However, we identify a critical support-limited bottleneck in this paradigm: on challenging reasoning tasks, the target model's samples often exhibit semantic redundancy, converging into the same erroneous "reasoning basins" that offer negligible reward contrast for policy updates.
To overcome this limitation, we propose a weak to strong learning paradigm, where a policy's exploration is informed by a weaker but computationally efficient auxiliary model. We introduce W2SPO, an off-policy RL method that injects short auxiliary segments, often as brief as 8 tokens, into intermediate target model trajectories, allowing the target model to complete the reasoning path from these diverted states. Policy updates are restricted to these short inserted segments based on final verifiable rewards.
Empirically, W2SPO achieves superior performance among evaluated 4B scale models on mathematical reasoning benchmarks, outperforming evaluated post-trained baselines. Compared with vanilla GRPO under the same sampling budget, W2SPO improves Pass@1 from 62.3% to 64.2% while achieving a 3.55 times training speedup. These results suggest that weak auxiliary branches can induce stronger target reasoning policies by expanding local exploration support.
Blogger's Review: This study significantly enhances reasoning capabilities in reinforcement learning by introducing weak auxiliary models, especially excelling in complex tasks. The use of short auxiliary segments for policy updates not only improves training efficiency but also strengthens the model's reasoning abilities, offering new insights and directions for future research.