In the realm of interactive agents, small language models have emerged as appealing backbones. However, direct distillation from robust teacher trajectories often translates rich multi-turn behaviors into one-shot imitation targets, which is inefficient in long-horizon environments where early decisions influence later states and rewards.
To address this, we propose Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task rewards. Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens within the replayed prefix, utilizing a policy-distilled SFT checkpoint to estimate their old log-probabilities. This unifies prefix learning and continuation learning within the same policy-optimization framework.
Experiments on TextCraft, BabyAI, and ALFWorld demonstrate that Prefix-GRPO enhances small-model agents over distillation and standard RL baselines, while ablation studies indicate that replay alone is insufficient without explicit prefix-token optimization. The implementation and reproduction scripts are available at GitHub.
Blogger's Review: The introduction of Prefix-GRPO significantly enhances the performance of small models in complex tasks by effectively leveraging teacher trajectories and optimizing the policy update process. This approach shows great potential in multi-turn interactions and deserves further exploration and application.