Abstract
On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored.
We identify two key inefficiencies in vanilla agent OPD:
- Full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision;
- Trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned.
To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision.
Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy-time frontier beyond vanilla OPD.
Blogger's Review: The introduction of TurnOPD presents a novel approach to long-horizon agent training, significantly enhancing training efficiency through a turn-aware budgeting strategy. It demonstrates superior accuracy and time utilization, especially in tackling complex tasks, making it a framework worth exploring and implementing in more applications.