Group‑based reinforcement learning has become a dominant paradigm for post‑training large language models, yet in multi‑turn agentic tasks with sparse terminal rewards it often offers only coarse credit for intermediate actions. Recent approaches such as GiGPO introduce step‑level advantages to provide finer credit, but these signals still rely on the final outcome of each individual trajectory, causing all actions in failed trajectories to receive the same negative credit even when some are actually useful.
We introduce Potential‑Guided Policy Optimization (PGPO) tailored for multi‑turn agentic tasks. PGPO first computes empirical state potentials from anchor‑state‑group return statistics within each rollout group. It then derives action advantages from the potential differences between adjacent states, enabling cross‑trajectory credit propagation. This yields finer‑grained step‑level credit, especially within failed trajectories where effective actions can be distinguished from erroneous ones.
Experiments on ALFWorld and WebShop demonstrate that PGPO achieves strong overall performance compared to recent group‑based RL methods. Additional analysis shows that PGPO provides more informative failure‑side credit signals with negligible training overhead.
Review