Training multi-turn evidence-reading agents solely with outcome-based reinforcement learning proves unstable due to the lack of direct credit for intermediate turns. In HotpotQA experiments using Qwen2.5-3B-Instruct, GRPO initially improves (standard F1 of 0.430) but subsequently collapses to 100% format-violating outputs. Diagnosis of training logs reveals a zero-advantage lock-in mechanism: all sampled trajectories incur the minimum format penalty (-2.0), group-relative advantages vanish, and policy-gradient loss becomes zero, resulting in an optimization deadlock.
To address this, we propose a variance-injection strategy: assigning per-turn rewards to intermediate evidence-reading turns prevents the group reward distribution from collapsing to a single value, thus preserving the variation required for GRPO's group-relative advantage. Contextual Information-Gain Policy Optimization (CIGPO) implements this strategy by using the marginal increase in the frozen reference model's log-likelihood of the ground-truth answer as the per-turn signal. With separate normalization for IG and F1 rewards and an IG-weight curriculum, CIGPO achieves a standard F1 of 0.518 on HotpotQA at the 3B scale (up from a base of 0.252; +105%), compared to 0.430 for the best GRPO checkpoint and 0.000 for the final GRPO checkpoint. CIGPO maintains meaningful reward variance and avoids zero-advantage lock-in throughout training. These results identify reward-variance collapse as a specific failure mode of outcome-only GRPO and demonstrate how turn-level IG rewards can prevent it in this HotpotQA setting.
Blogger's Review: The CIGPO approach effectively resolves the optimization deadlock of traditional GRPO methods by introducing turn-level rewards, providing a new perspective for multi-turn evidence reading tasks. Future research could explore the application of this strategy in more complex environments to enhance the learning efficiency and effectiveness of agents.