Reinforcement Learning from Verifiable Rewards (RLVR) drives large reasoning models, yet it is prone to a "zero-reward collapse" where a sudden rise in task difficulty wipes out the learning signal. Discarding these failed trajectories wastes valuable diagnostic clues about the model's cognitive gaps that could aid recovery. Existing approaches typically fall back to static, off‑policy interventions such as pre‑defined hint libraries or post‑hoc fine‑tuning, lacking adaptive, forward‑looking guidance to escape the collapse. We therefore shift to an on‑policy pedagogical paradigm and introduce MENTOR, a multi‑agent framework that acts as a live mentor. MENTOR diagnoses the root cause of failure in real time and dynamically generates targeted instructional prompts to steer the learner’s next attempt. As a plug‑and‑play module, MENTOR integrates seamlessly into existing policy‑optimization pipelines like GRPO without altering the training infrastructure. Extensive math and code generation experiments show that MENTOR markedly mitigates zero‑reward collapse and outperforms state‑of‑the‑art baselines. Notably, on Llama‑3.2‑3B‑Instruct it achieves up to a 78.3% average relative gain over the base model.
Review