In this study, we investigate on-policy distillation (OPD) for agentic tasks, where a large language model (LLM) agent interacts with an environment over multiple turns, and a student imitates a teacher's multi-turn interaction history. Traditional online OPD is costly, requiring fresh student rollouts through the environment and teacher queries at visited histories for each update.
To address this, we propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps while the teacher provides dense per-step supervision without executing new environment interactions.
We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes.
Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout than OPD. Thus, ReOPD turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
Blogger's Review: The introduction of ReOPD presents an innovative solution to the high cost of online policy distillation by significantly enhancing efficiency through the reuse of teacher trajectories. Its potential in multi-turn interaction scenarios is particularly noteworthy, and balancing the reliability between student and teacher will be a key focus for future research.