Long‑horizon agents must execute workflows where later steps depend on earlier outputs and shared state changes. The main evaluation challenge is to enlarge task requirements and their dependencies while keeping tasks runnable and outcomes verifiable. We introduce SkillHorizon, a benchmark that builds long‑horizon tasks by composing skills across seven business environments. Its generators apply domain‑specific rules to turn reusable skill procedures into concrete business activities, linking them through input‑output dependencies, shared resources, and state updates, so that workload and coordination demands grow together. Successful reference executions confirm task feasibility and measure interaction workload; programmatic evaluators assess both local goal completion and whole‑task success. We evaluated five language models on 1,050 tasks with interaction workloads ranging from 64K to 1M tokens. Mean local‑goal completion dropped from 90.4% to 46.8% as workload increased, indicating that completed work could not keep pace with expanding requirements. At larger scales agents more frequently submitted unfinished final answers while still having tool‑call allowance left. Trace analysis also revealed cases where scripts ran correctly but their outputs failed downstream business checks. SkillHorizon offers a configurable testbed for studying sustained progress, dependency coordination, and complete task delivery in growing workflows. The code is available at https://github.com/KnowComp/SkillHorizon
Review