ExpertCrowd‑SWE (EC‑SWE) is a benchmark of one hundred expert‑authored, multi‑file software engineering tasks designed to evaluate autonomous coding agents operating in large, unfamiliar codebases. Each task pairs a change request written by a practising engineer with a pinned commit of a real repository, a sealed reference solution, and a task‑specific grader. The grader uses weighted, mostly executable criteria to award partial credit; must‑pass blockers and pass/fail gates encode the decisions a maintainer would reject. The tasks span eighty‑seven open‑source projects and eleven programming languages. The median ticket length is 148 words, while the median reference solution touches 471 lines. Every task is calibrated before release so that the reference solution scores 1.0 and both the untouched repository and a no‑op change score 0; roughly half also include lazy or adversarial control patches that the grader must classify correctly. The final hundred tasks were selected for difficulty against current frontier models. We evaluated seven frontier models, five runs each, totaling 3 495 scored runs, on a single platform harness that supplies each model with its provider‑native tool interface. Claude Opus 5.5 achieved 67.3% and GPT‑6 Astra 63.7%, followed by Claude Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0%), and a statistically indistinguishable group of Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 (32‑34%). No task is solved by every model, and on eighty tasks no model earns full credit on all five runs. Scores span thirty‑five points. Depending on the model, agents take a median of thirty‑three to one‑hundred‑sixty‑six steps per run, with little correlation to score. We release twenty‑four tasks together with their environments, graders and results, while holding out seventy‑six to limit contamination.
Review: EC‑SWE offers a fine‑grained, quantitative benchmark for measuring large‑model coding ability on real, multi‑file changes, exposing current weaknesses in cross‑file reasoning and blocker handling, and providing clear targets for future model improvements.