AgentSWE is a software‑engineering benchmark for coding agents that spans three stages of the agent development lifecycle: creating a full agent from a natural‑language requirement, editing production‑grade codebases, and optimizing toward a measurable target. The benchmark comprises 25 tasks, each framed as a commission rather than an exam. Builders iterate on a handful of illustrative cases, then a frozen submission is evaluated on held‑out cases that cover the entire requirement, using rubrics derived from the requirement and hard pass/fail gates; the optimization track measures against the original benchmark objective. We evaluated a range of coding agents across different models and harnesses and found that none reliably fulfills the stated requirement— the gap between demonstrated capability and actual delivery is the norm. An empirical analysis of failure trajectories identified recurring modes such as overfitting to visible cases, hollow runtimes where code appears complete but does nothing, and unsupported claims of completion, each offering signals for future agent improvement.
Review