Large language model (LLM) agents are increasingly deployed in long‑horizon environments where early mistakes can cascade into dead‑end trajectories. To cope, agents must proactively recover from such states and turn abandoned exploratory experience into guidance for future decisions.
We introduce ReTRAIL, a framework that lets the environment rewind to earlier states while preserving valid progress and carrying forward knowledge learned from discarded branches. Concretely, the environment state moves backward, whereas the agent’s knowledge continues to advance.
ReTRAIL trains a shared policy with two complementary reward signals:
- State‑Anchored GRPO: compares alternative continuations from revisited states to improve execution quality.
- Branch‑Advancement Reward: rewards reflective behavior based on the progress of subsequent exploration, encouraging the agent to leverage insights from abandoned branches.
Experiments on four interactive benchmarks (MineSweeper, Sokoban, ALFWorld, ScienceWorld) show that merely adding a rewind primitive yields limited benefit and can even degrade performance. In contrast, ReTRAIL enables agents to rewind from unproductive states and use knowledge from discarded branches to steer later exploration, markedly improving success on long‑horizon tasks.
The code is publicly available at https://github.com/whi497/ReTRAIL
Review