Computer‑use agents can accomplish complex tasks, prompting increasing reliance on automatic judges to decide success, either for training or fully automated evaluation. Yet their reliability on long‑horizon tasks that span multiple applications remains uncertain. A trajectory—comprising hundreds of screenshots and actions—may look complete but can violate instruction constraints or cause unwanted side effects. A judge must examine the entire trajectory against the user’s instruction to spot such errors. To address this, we introduce AgentHorizon, a benchmark containing 1,373 instruction‑trajectory pairs collected from 166 hours of human recordings across three operating systems. By recording trajectories for closely related instructions, we can generate negative tasks by swapping the instructions; this paired design directly tests a judge’s ability to distinguish a successful trajectory from one that fulfills a similar yet incompatible request. The benchmark is released in three splits: a frontier split (AgentHorizon, AH), a simplified split (AgentHorizon‑Simple, AH‑S), and a development split (AgentHorizon‑Development, AH‑D). We evaluate eleven judges in two modes: (1) feeding the full trajectory (up to 300 screenshots and actions) directly, and (2) using them as coding agents within five agent harnesses. The best agentic judge, GPT‑5.5, achieves 80.9% balanced accuracy on the AH subset. Tool‑use improves certain models but degrades performance for open‑weight models, and judges differ dramatically in accepting valid trajectories while rejecting failed ones. These findings highlight the need for judges capable of locating and verifying often hidden evidence of proper task completion within long interaction histories.
Review: AgentHorizon offers a systematic evaluation platform for automatic judges on long‑horizon computer‑use tasks, exposing current models’ shortcomings in evidence detection and constraint adherence, and pointing to clear directions for future improvements.