Reinforcement learning has been employed to boost the reasoning ability of large language models, yet it still struggles with long‑horizon tasks due to two major bottlenecks: (1) high task complexity makes successful trajectories scarce, slowing convergence; (2) reliance on sparse final rewards forces uniform updates across all steps, encouraging redundant actions. When both the quantity and quality of successful trajectories cannot be guaranteed, overall performance suffers. This stems from existing RL frameworks that depend entirely on the base model’s reasoning capacity and lack active trajectory optimization.
To address these issues, we introduce TrajEvolve, a trajectory‑evolution paradigm equipped with hindsight credit assignment. In addition to the trajectories inferred by the model, TrajEvolve generates “evolutionary trajectories” by refining low‑quality steps from historical runs, exposing the policy to high‑quality compositions that lie within the base model’s support but are rarely sampled under typical training budgets.
Moreover, TrajEvolve reformulates step‑level continuous credit assignment as a binary classification problem: whether a step is necessary for task completion. This simplification reduces learning difficulty and enables step labels to be obtained via counterfactual verification, achieving precise identification of redundant steps.
Extensive experiments on two public datasets demonstrate that TrajEvolve outperforms conventional RL methods in success rate, convergence speed, and generation quality.
Review: By evolving trajectories and using binary credit discrimination, TrajEvolve effectively mitigates sample scarcity and redundancy in long‑horizon tasks, offering a practical new direction for reinforcement learning with large models.