Coding agents are shifting from handling a single request at a time to acting as persistent assistants, where users can stream builds, data‑processing jobs, and bug fixes to the same machine. An agent must decide when to accept new tasks, how to delegate and overlap them, and stay within CPU and memory limits.
Existing benchmarks only evaluate isolated tasks and cannot measure this workflow. We introduce AOS‑Bench, which treats an agent system as a workload manager: verified tasks arrive over time and must be completed under shared CPU and memory constraints, while the same tasks run one‑by‑one serve as a per‑system reference.
Across five model‑harness pairs, isolated task success does not translate evenly to stream execution. Under heavy workloads every system loses 13‑23 points; under medium workloads isolated success spans 11 points but stream success spans 26 points, and two systems that differ by 0.6 points in isolation deliver 54% and 68% of the same tasks.
Policy analysis shows that task intake is the main bottleneck: serial dispatch leaves 73% of heavy‑workload tasks unclaimed. More elaborate management does not necessarily help— a fixed concurrency of two stays within four points of the best agent‑managed policy, and under every policy median CPU usage remains below 30% of provisioned cores while tasks wait.
These findings identify workload management as a distinct capability for agents, and AOS‑Bench as a way to measure it.
Review