In disaggregated LLM serving, the prefill stage is compute‑heavy while the decode stage is memory‑heavy, and they are placed on separate GPU pools. Systems such as DistServe, Splitwise and Mooncake enable fast separation, but routing still decides which instance handles each request.
This paper studies a router that estimates the additional completion time on each instance using the exact prompt length, predicted output length, post‑admission KV‑cache pressure and SLO class. The policy is implemented in a discrete‑event simulator and validated on eight NVIDIA A40 GPUs (each running a vLLM engine) with NIXL transferring KV caches between pools, all workloads running at measured saturation.
Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput of 0.864, outperforming round robin, least loaded and a length heuristic (0.835‑0.847) and showing the lowest variance across traces. It beats round robin and the length heuristic on all three traces, beats least loaded on two, and trails by only 0.003 on the third, which is within run‑to‑run noise. Hardware calibration matters: constants derived from the simulator add 4.5 goodput points and roughly 40 % of the tail‑latency advantage, otherwise the scorer collapses to simple queue counting.
Benefits grow with decode‑pool size and traffic heterogeneity but disappear in pools with only three instances, where queue counts are often sufficient. Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, while blind spreading performs better. With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.
Review: The study demonstrates that fine‑grained hardware calibration can substantially improve routing decisions, offering a practical path to higher resource efficiency in disaggregated LLM serving.