This paper examines the choice between teacher‑generated demonstrations and student‑generated experience when distilling interactive LLM agents, traditionally framed as off‑policy versus on‑policy. We argue that the distinction is hierarchical: the policy that creates the interaction history need not be the same as the policy that supplies the token prefix for the current action. A hierarchical distillation framework is introduced to formalize these two levels of on‑policy‑ness, allowing explicit control over their pairing, mixture ratios, and training schedule.
Experiments use self‑distillation where a frozen teacher has access to task guidance unavailable to the student, and a three‑stage controlled study is conducted across three models and four interactive environments. Findings show that histories induced by the student combined with teacher‑generated action prefixes generally give the strongest single training signal, while following the student at both levels performs worst. Purely on‑policy data become beneficial when mixed with other configurations. Even when teacher/student proportions are identical at each level, different pairings lead to substantial performance gaps, indicating that pairing matters more than the total amount of on‑policy data. Moreover, front‑loading fully teacher‑generated contexts and delaying fully on‑policy contexts outperforms a static mixture. These results offer practical guidelines on where, how much, and when agent distillation should follow the student.
Review