RLVR improves language‑model reasoning but usually relies on independent stochastic rollouts for exploration, which often resample similar failing strategies. Prior work has introduced dependence by feeding incorrect solutions into the context of subsequent attempts, yet this raises the question of how to assign credit to each try. We reinterpret credit assignment as a Stochastic Shortest Path problem in classical reinforcement learning, aiming to minimize the expected cumulative cost to reach a terminal state—a verified successful response. Under reasonable simplifying assumptions this perspective leads to a straightforward modification of GRPO, which we call Retry‑Aware Policy Optimization (ReAPO). ReAPO allocates extra generation steps to failed trajectories, allowing successful retries to be reinforced without rewarding preceding failures, and assigns credit to each attempt in a theoretically principled manner. Experiments span Qwen3 and OLMo‑2 models from 0.6B to 4B and eight mathematical and scientific reasoning benchmarks. ReAPO consistently outperforms GRPO and recent baselines on first‑attempt success, and ablations confirm the benefits of serial sampling and ReAPO’s credit assignment. On Qwen3‑0.6B, ReAPO beats GRPO even with twice the group size or twice the training steps, reaching peak accuracy with 39% and 31% fewer generated tokens respectively. Overall, ReAPO offers a simple yet effective way to boost LLM reasoning performance through improved exploration and retry‑aware credit allocation.
Review: The key insight of ReAPO is treating retries as valuable exploratory steps and grounding credit assignment in the stochastic shortest‑path framework, enabling faster convergence to correct answers under limited computational budgets.