Reinforcement learning (RL) has become a widely adopted technique for improving large language models (LLMs) on complex tasks. However, existing RL methods face challenges in training agents with longer-horizon interactions. A major bottleneck is distinguishing the contribution of different actions in long-horizon interactions, leading to high optimization variance.
To address this, we introduce a novel policy gradient method, Hindsight Policy Optimization (HPO), which projects both the current policy distribution and the hindsight distribution into an intent space and extracts low-variance learning signals from the Wasserstein distance between them.
We theoretically and empirically demonstrate that aggregating semantically similar states and actions in the intent space yields a bounded-variance estimator and improves policy performance stably.
Our code is available online.