In the field of artificial intelligence, agents have become capable of autonomously completing well-defined short tasks. However, existing terminal benchmarks primarily focus on simple problems that finish within minutes and are evaluated solely based on their final outcomes. This setup overlooks intermediate progress and partial solutions, resulting in sparse reward signals and an incomplete understanding of agent capabilities. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark comprising 46 long-horizon tasks across nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, further decomposed into fine-grained graded subtasks. This design allows for dense intermediate rewards and partial credit, enabling evaluations to capture not only whether an agent reaches the final goal but also how far it progresses in open-ended workflows.
Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluated 15 frontier models and found that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves a pass rate of 15.2% at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal room for improvement. We further analyze failure modes and error patterns and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
Blogger's Review: This new benchmark not only expands the evaluation approach for agents in complex tasks but also emphasizes the importance of intermediate progress, which is likely to drive more effective long-term planning and enhancement of agent performance.