In reinforcement learning, particularly in post-training phases for large language models, credit assignment remains a challenge. Agents typically receive sparse, trajectory-level rewards only at the end of an episode, making it difficult to discern which intermediate actions contributed to success or failure.
To propagate delayed outcomes back to individual decision steps without relying on costly auxiliary value models, we introduce Generalized Advantage Grouped Policy Optimization (GAGPO).
GAGPO is a critic-free reinforcement learning method designed for precise, step-aligned temporal credit assignment. It constructs a non-parametric grouped value proxy from sampled rollouts and uses it to compute TD/GAE-style temporal advantages, recursively propagating outcome supervision backward through time.
Combined with group-wise advantage normalization and an action-level importance ratio, GAGPO extracts stable, localized optimization signals directly from multi-turn trajectories.
Experiments on ALFWorld and WebShop demonstrate that GAGPO outperforms strong reinforcement learning baselines. Further analyses reveal faster early-stage learning, improved interaction efficiency, and smoother optimization dynamics, suggesting that GAGPO offers a simple yet effective framework for multi-turn agentic reinforcement learning.
Blogger's Review: GAGPO's innovative approach to temporal credit assignment significantly enhances the performance of multi-turn reinforcement learning. Its critic-free design opens new avenues for efficient learning and is worth attention!