NeFut Logo NeFut
中 Admin Login

[CS.AI] Agent Architecture Cost-Success Trade-offs: A Controlled Evaluation across Enterprise-Style SOPs

Published at: 2026-11-11 22:00 Last updated: 2026-11-12 00:51
#AI #Machine Learning #LLM

Enterprises are increasingly deploying large language model agents to carry out standard operating procedures (SOPs), which are typically long, branching, and tool‑intensive. Yet there is little controlled evidence to guide the choice among competing agentic architectures. Holding the model, tasks, and tools constant and varying only the architecture, we compare four designs—function‑calling single agent, skill‑augmented single agent, centralized multi‑agent orchestration, and a triage swarm (eight variants)—across 13 enterprise‑style SOPs, and introduce a skill‑definition quality rubric to build matched high‑ and low‑quality skill corpora. Three takeaways emerge. First, there is no universal best architecture; the optimal choice depends on the SOP. Success rates on a given SOP range from below $10\%$ for the hardest to near $100\%$ for the easiest, and overall no architecture is statistically separable, with added structure only showing a directional (not yet significant) benefit on complex, branching SOPs. Second, added architectural cost must be justified by a gain in task success. Every coordinated arm (multi‑agent or skill‑augmented) consumes $1.4$–$4.7\times$ the tokens and $2.6$–$5.8\times$ the latency of the single agent, yet the gains are usually small and rarely significant, making the single agent the efficient default; high‑quality skills raise success to $73.1\%$ from $67.4\%$, while degraded skills fall below it. Third, implementation quality outweighs the single‑ versus multi‑agent decision. A plausible multi‑agent implementation still loses to the single agent, but a targeted redesign of it becomes the only arm to significantly beat the baseline ($75.6\%$ vs. $67.4\%$).

Review: For enterprise SOP automation, architecture selection should prioritize task complexity and implementation quality rather than defaulting to multi‑agent or elaborate orchestration, which often incur disproportionate cost for marginal benefit.

Original Source: https://arxiv.org/abs/2611.12953

[h] Back to Home