LLM scheduling is critical for service, yet existing designs' suitability for agentic service remains unclear. The workload changes in two key ways:
- Agents act only on complete responses, making tokens per second (TPS) the primary goal while relaxing per-token latency requirements.
- KV reuse exceeds 80% of request tokens in a production trace from BAILIAN, compared to 54-62% in chat.
This paper provides a systematic study of request scheduling for agents based on two real-world traces. We found that existing schedulers overly prioritize routing requests to instances caching their KV, leading to overload on a few instances while leaving others idle, capping TPS. Thus, we present two key insights:
- Load balancing need not sacrifice all KV reuse, thanks to the global-tier KV store.
- By utilizing intra-session locality, balancing a small fraction of requests—the first request in each agent session—is sufficient to balance the cluster without sacrificing most KV reuse on local instances.
SMETRIC realizes these insights with balanced session-centric scheduling: it routes each session's first request purely for load balance, while follow-up requests are handled in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from user inputs, keeping the scheduler clean and stateless. SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation compared to state-of-the-art schedulers, also achieving better per-token latency.
Blogger's Review: The SMETRIC study showcases innovative scheduling strategies in agentic service, effectively enhancing system performance through balanced session-centric approaches. Its ability to find a balance between load balancing and cache reuse provides a new direction for future LLM services, making it worthy of attention and emulation.