In long-horizon interactive tasks, large language model (LLM) agents have demonstrated strong decision-making capabilities, yet they struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents.
PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization.
Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80% of the tasks. In the Minesweeper environment, PivoARL improves over GiGPO by more than 45% and reduces interaction turns by about 42% on average compared with full-retry methods.
Code is available at GitHub.
Blogger's Review: The local retry mechanism and pivotal-aware credit assignment proposed by PivoARL provide innovative strategies for enhancing LLM agent decision-making capabilities. Particularly in complex tasks, it significantly reduces interaction costs and showcases the potential for efficient experience utilization. Its excellent performance across multiple benchmarks suggests a promising future for applications in intelligent agents.