Long‑horizon interaction requires agents to make a series of decisions in a continuously changing environment. Outcome‑based GRPO provides only a global reward signal and lacks fine‑grained guidance for intermediate actions. Recent work distills complete trajectories into reusable skills and employs hindsight‑conditioned OPD to assign dense token‑level credit. However, the probability shifts induced by skills do not necessarily align with reward‑based credit, especially when an erroneous action alters subsequent states, causing skill‑conditioned supervision to mismatch the current context.
To address this mismatch we introduce Error‑Truncated Hindsight Distillation (\methodname{}). The method routes two complementary hindsight skills according to trajectory outcomes and error boundaries: Positive skills capture successful workflows to reinforce whole successful trajectories and decisions made before the error; Negative skills capture failure‑specific corrections, targeting decisive mistakes. The post‑error suffix is optimized exclusively by GRPO, preserving outcome‑based learning while avoiding potentially mismatched hindsight supervision.
Extensive experiments on several long‑horizon benchmarks show that \methodname{} achieves higher overall task success rates and more effective fine‑grained credit allocation compared with baselines that rely solely on GRPO or a single hindsight skill.
Review