In LLM (Large Language Model) serving, the key-value (KV) cache has emerged as a primary memory object rather than merely a temporary tensor per request. This survey classifies over thirty KV management systems and frameworks along four axes: locality, lifetime, ownership, and substrate. These axes reveal five architectural archetypes — local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier.
Once the workload and hardware are fixed, ownership accounts for much of the design variance among distributed systems. Additionally, the survey audits current evaluations and identifies seven missing KV-specific measurement metrics, linking them to open issues in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.
Blogger's Review: This in-depth survey on KV cache management provides a systematic perspective for LLM serving, helping to understand the strengths and weaknesses of different architectures and their applicable scenarios. In the context of the growing demand for efficient memory management in AI, the identified seven measurement metrics point to future research directions.