Reinforcement learning has emerged as a promising paradigm for enhancing large language model (LLM) agents in long-horizon search tasks, where agents must make a series of intermediate decisions before obtaining a final outcome. However, existing methods face a key limitation: rollout budgets are often allocated without explicitly assessing the utility of intermediate states. Consequently, substantial computation may be wasted on low-value states, even though different branches can vary significantly in their informativeness.
To address this, we propose Information Gain-based Rollout Policy Optimization (IGRPO), a policy optimization framework that treats intermediate-state informativeness as the organizing principle for rollout collection. Specifically, IGRPO performs budget-aware tree-structured rollouts by allocating expansion budgets according to node-level informativeness, allowing more informative branches to be expanded more frequently while progressively suppressing less promising branches.
We further demonstrate that the information gain-based rollout induces an explicit limiting teacher distribution over trajectories, which naturally yields a clear policy optimization target, thereby unifying adaptive tree-structured exploration with principled policy learning under a single framework. Experiments on seven challenging search-augmented QA benchmarks show that IGRPO consistently outperforms strong baselines under the same rollout budget constraints, validating the effectiveness of leveraging the induced teacher distribution to guide policy optimization for long-horizon search agents.
Blogger's Review: The proposed IGRPO offers a novel approach to optimizing rollout strategies by leveraging information gain, enhancing the decision-making process of LLMs in long-term tasks. This effectively utilizes the informativeness of intermediate states, significantly improving search efficiency with important theoretical and practical implications.