Multi-agent systems (MAS) built from language models assume collaboration outperforms a single-agent system (SAS), yet the conditions for benefit remain unclear. A prior systematic study introduced a capability‑saturation rule: coordination helps while the SAS is weak and hurts once it becomes strong, with the crossover near 45% single‑agent accuracy. That rule was derived on fully autonomous benchmarks without user involvement.
We evaluate the rule in a user‑conversation setting under a written policy, using the dual‑control benchmark $\tau^2$-bench across airline, retail and telecom domains. Six models from three families are compared: the stock SAS and a centralized MAS built from the same model, covering roughly 7,950 simulated conversations and recording one completion‑token ledger per episode. The study was pre‑registered with a public change log; the only unmet validity bar was that the MAS planner never saw the tool schemas.
With a fixed simulated user, no cell shows a MAS advantage: out of 13 valid cells, MAS is worse in 12 and tied in one, and the fitted gain is negative across the observed capability range (weighted mean -28 points, 95% CI [-44, -12]). In the two cells where the rule predicts a benefit, MAS loses by 13 and 22 points respectively; without a budget ceiling it at best draws level at 5.5 times the SAS's token spend. A single gain appears near the 45% crossover under a second simulator. Whether the gain declines with capability is unresolved (slope -0.41, 95% CI [-1.51, +0.68]), as model‑by‑domain effects dominate. MAS does not noticeably reduce wrongful database writes; it converts other failures into budget exhaustion and inaction.
The takeaway is that for agents serving people, SAS remains the default; a multi‑agent design must provide compute‑matched evidence from the setting itself. All trajectories, ledgers, prompts and code are released.
Review