NeFut Logo NeFut
中 Admin Login

[CS.AI] ServerBench: How Coding Agents Can Design Specialized LLM Serving Systems

Published at: 2026-11-06 22:00 Last updated: 2026-11-07 12:32
#AI #optimization #LLM

Large language model (LLM) coding agents create software by trial‑and‑error, opening the possibility for them to design the serving infrastructure that runs their own code. An effective serving stack must reconcile implementation choices with session dependencies, latency limits, and correctness guarantees. ServerBench is introduced as a benchmark that tasks agents with building complete, specialized LLM serving systems. All agents optimize a single metric—dollars per dollar: the token value qualified by latency per dollar of GPU cost. The benchmark preserves session state via replay and employs an independent verifier to check the submitted system. Across 11 coding‑agent models and three deployment settings, Claude Sonnet 5.5 reaches 2.97 dollar per dollar on Gemma E4B, a 4.87× improvement over the vLLM reference; 22 of 33 runs produce verified systems, and 13 exceed the strongest baseline. Three insights emerge: high‑value designs align capacity upgrades with load selection; agents interpret performance feedback through workload‑relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimizations. ServerBench thus enables systematic study of how agents translate service objectives and feedback into effective, verified serving systems for given tasks.\n\nReview

Original Source: https://arxiv.org/abs/2611.06841

[h] Back to Home