SWE-Shift is a benchmark designed to evaluate coding agents under mid‑execution workspace changes. Most repository‑level benchmarks assume that tasks and environments stay static during execution, but real software development frequently sees code, dependencies, runtime state, and requirements evolve while an agent is still working.
The benchmark creates two parallel continuations from the same saved history and workspace:
- Clean continuation – proceeds without any intervention, keeping the original task unchanged.
- Shifted continuation – receives a task‑relevant update at a scheduled point (e.g., dependency upgrade, file modification, tool/session swap).
Success in a Shifted scenario requires the agent to:
- Complete the current task;
- Preserve work that remains valid;
- Satisfy the updated state and evidence checks.
SWE-Shift comprises 120 scenarios across 12 change families (such as tool changes, stale evidence, runtime state invalidation, etc.), each evaluated at nine intervention positions. We tested 17 coding‑agent systems; all showed lower success rates on Shifted continuations than on Clean ones. For instance, GPT‑6 Astra dropped from 67.35% to 24.79%, with a paired success rate of only 20.34%. Tool/session changes and stale‑evidence cases yielded zero paired success for every system. Late‑position paired success varied: some systems declined further, while others improved.
These findings demonstrate that high performance on unchanged tasks does not guarantee the ability to preserve work, recover state, or refresh evidence after an unexpected update.
Review