Abstract
Long-horizon tasks require sustained perception, reasoning, and exploration, posing a persistent challenge for large language model (LLM) agents. This gap is reflected in their limited performance on continual learning benchmarks like ARC-AGI-3, especially when models are evaluated out of the box. Various agent frameworks have been proposed to close this gap, each committing to a strategy for handling long sequences of observations, specifically regarding what information to save from the environment and how to load it into model context, a choice we argue is particularly consequential.
Existing methods for context management face a significant tradeoff: preserving more information makes retrieving relevant details less tractable. We propose PRO-LONG, a minimal context management framework built around programmatic memory for LLM agents in long-horizon exploratory settings. PRO-LONG addresses this tradeoff by maintaining a complete, structured interaction log and leveraging recent advancements in coding agents to search this history efficiently.
On the full ARC-AGI-3 public game set, PRO-LONG improves upon a base coding agent by an average of 18.0 percentage points across frontier models, and matches or exceeds state-of-the-art specialized harnesses (up to 76.1% pass@1) while using 4.2-5.8x fewer tokens. With Fable 5, PRO-LONG achieves 97.4% best@2 at a total cost of $1,750. Relevant code and logs are available at GitHub.
Blogger's Review: The introduction of PRO-LONG offers a fresh perspective on managing long-horizon tasks for LLM agents, particularly addressing the tradeoff between information management and retrieval efficiency. The use of a programmatic memory framework significantly enhances model performance, warranting further exploration and validation in practical applications.