Long-Horizon-Terminal-Bench (LHTB) measures whether an agent can keep making progress through extended workflows, not just whether its final state passes a binary check. LHTB 2.0 preserves the original 46 tasks, fixes task and evaluation‑infrastructure bugs, and adds 31 new tasks, resulting in 77 tasks across 13 domains and eight task types.
The new tasks are generated by a systematic reconstruction‑task pipeline: start from a working scientific or engineering artifact, use a parameterized reference implementation to create multiple scenarios, expose six scenarios as behavioral evidence, hold out three for evaluation, and apply dense progress ladders to reward partial progress while preventing memorization of public outputs.
The evaluation horizon is extended from 90 minutes to three hours, with verifier checkpoints recorded every 30 minutes, enabling direct measurement of sustained progress within a single run.
Across 1,145 scored model‑task runs from 15 frontier models, GPT‑6‑astra leads with a dense mean reward of 0.69 and achieves R≥0.95 on 37 tasks; GPT‑5.6‑sol follows with a dense mean reward of 0.67 and 32 tasks meeting the same threshold. For the 30 reconstruction tasks, binary verifier scores average 0.221, whereas dense ladder scores average 0.653, showing that outcome‑only evaluation discards substantial intermediate progress.
LHTB 2.0 offers a broader, hardened, and process‑aware testbed for studying reliable agent execution over hours.
Review