Scaling the training of large language model (LLM) agents usually requires a large set of executable environments, and the construction, maintenance, and operation of these environments limit the expansion of training experience. Offline trajectories offer a way to reuse interaction data, yet refining the current policy without revisiting the environments remains difficult: supervised fine‑tuning (SFT) can cause catastrophic forgetting, while online reinforcement learning needs real‑time interaction. To tackle this, we propose Step‑Advantage Policy Distillation (SAPD), which learns from offline trajectories using relative step advantages. SAPD decomposes multi‑turn trajectories into single‑step decision contexts for on‑policy distillation, allowing the student model to generate responses conditioned on recorded histories and introducing relative step‑advantage signals to stabilize training. The whole process requires no new environment observations and no additional multi‑turn interactions. Experiments on ALFWorld, WebShop, and search‑based QA show that SAPD improves performance across various teacher configurations, mitigates catastrophic forgetting compared with SFT, and runs at least $4.7\times$ faster per training step than online OPD. The code will be released soon.
Review: SAPD’s use of relative advantage signals provides stable offline distillation, balancing efficiency and forgetting mitigation, marking a significant step forward for offline learning of large models.