LLM inference has become an operational workload that must be provisioned, monitored, and billed against latency SLOs. A single throughput metric cannot answer the real questions operators face because generative serving couples a prefill phase and a decode phase with very different service characteristics, continuous batching reshapes the effective service process, and user‑visible tail latency can degrade long before aggregate throughput saturates. GAUSS addresses this by pairing calibrated, phase‑specific latency surfaces for prefill and decode with a Lindley‑recurrence queueing model. It predicts the full distributions of time‑to‑first‑token (TTFT) and time‑between‑tokens (TBT) and estimates the sustainable request rate for a given workload, engine, and accelerator. Instead of replaying a single trace, GAUSS uses ensemble Monte‑Carlo sampling to evaluate performance under stochastic arrivals. This enables reasoning about unseen workloads, routing policies, and hardware, and captures latency tails that trace‑driven simulators systematically under‑sample. We illustrate how the model supports operating‑point selection, traffic segmentation, and shortest‑remaining‑processing‑time (SRPT) scheduling capacity questions that traditional trace‑driven simulators cannot address.
Review