Prefix caching enables Large Language Model (LLM) servers to reuse KV states, speeding up the prefill stage for requests that share a prompt prefix and thus reducing latency and serving cost. Because memory is limited, the KV cache eviction policy becomes the key factor determining the overall benefit of prefix caching. Production traffic may consist of single‑turn, multi‑turn, agentic, or any mixture of these request types. Our experiments show that agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency. Across 20 workload mixes, no single policy is universally best: vLLM’s default LRU yields 12% more cache hits than the frequency‑oriented S3FIFO on agentic‑heavy mixes, yet trails by up to 22% elsewhere, and even a single workload parameter can flip the winner for a given mix. We therefore introduce WAKMan, a workload‑aware KV cache manager that selects its eviction policy from live traffic without knowing request types or reading prompts. Results indicate that WAKMan improves hit rate over LRU by an average of 5 percentage points and achieves the lowest mean and worst‑case optimality gap among all policies.
Review