As coding agents become more capable, developers increasingly rely on them to fix bugs through multi‑turn interactions. Existing benchmarks mostly present a complete issue report in a single turn, lacking interaction, which leads to false positives and false negatives—incorrect patches may be accepted and correct ones rejected. To address this, we introduce UserSWE, a framework that evaluates coding agents’ ability to fix repository‑level bugs via multi‑turn conversations with dynamic users, providing execution‑grounded verdicts after each turn. The construction pipeline transforms repository histories and existing single‑turn benchmarks into multi‑turn tasks. At each turn, an information pool restricts what the simulated user can reveal, and a persona chain sampled from a transition model calibrated on 20,859 development sessions governs the user’s language. After every turn, tests are run; a pair of agents from different model families writes and gates each verdict, using test results as primary evidence and model judgment only for aspects the tests cannot resolve. We evaluated eleven agents on 72 tasks from 42 repositories and observed that rankings change with the interaction budget: Claude Fable 5.1 solves the most tasks in one turn, GPT‑6‑Astra solves the most by the eighth turn, and Claude Opus 5 leads at most intermediate budgets. These findings indicate that single‑turn performance is insufficient to characterize interactive coding ability.
Review