Training multi‑turn LLM agents with reinforcement learning usually relies on trajectory‑level rewards that give each step the same advantage, making it hard to pinpoint which decisions caused success or failure. Self‑distillation methods add privileged information for finer supervision, yet most apply the same type of privilege uniformly, overlooking a key asymmetry: routine steps need little guidance while critical error steps require corrective direction beyond what environment feedback offers.
AHEAD introduces a step‑aware framework that aligns different supervision sources with different step types. The teacher receives dense environment feedback on all steps and, on error steps, also obtains LLM‑generated corrective hints to supply missing direction. The modification to the standard GRPO algorithm is minimal, adding conditional hints only during teacher updates.
Experiments span ALFWorld, WebShop, and search‑based QA across three model scales. AHEAD raises task success by 13.3 points on ALFWorld and 11.0 on WebShop for a 7B model compared to GRPO, reaches the same success rate in fewer training steps, and solves tasks under tighter interaction budgets, outperforming outcome‑only RL and prior self‑distillation baselines.
$$ R_t = \sum_{i=t}^{T} \gamma^{i-t} r_i $$
Blogger's Review: AHEAD’s distinction between routine and error step supervision provides precise guidance for LLM agents, revealing a promising direction for self‑distillation in reinforcement learning.