In the realm of large-scale online reinforcement learning (RL), effectively enhancing long-term reasoning and tool usage capabilities poses a significant challenge. While multi-teacher on-policy distillation (MOPD) offers a novel approach for knowledge distillation, it still requires coupled inference and backward passes, limiting scalability and computational efficiency. To address this, we propose REGEN: Replay-recycling for Expert-to-Generalist Distillation.
The innovation of REGEN lies in its approach of not relying on multiple teacher models for distillation. Instead, it recycles the replay memory generated from the specialized training of teacher models and employs offline RL algorithms for training. This method completely decouples the rollout sampling from the backward training process, significantly reducing training costs.
Across tasks such as mathematical reasoning, code generation, and instruction following, REGEN matches the accuracy of MOPD at a substantially lower cost. Ultimately, REGEN has the potential to transform online RL into a data synthesis process rather than a one-off learning stage, and it can be scaled for large-scale post-training without requiring heavy computational loads.
Blogger's Review: REGEN significantly enhances the efficiency and cost-effectiveness of reinforcement learning through its innovative replay-recycling mechanism, showcasing broad applicability in multi-task learning. Its methodology not only reduces computational resource demands but also opens new avenues for future reinforcement learning research.