NeFut Logo NeFut
Admin Login

[CS.AI] vToken: Token-Level Virtualization for Reclaimable KV Caches

Published at: 2026-08-15 22:00 Last updated: 2026-08-16 07:03
#AI #Machine Learning #optimization

We present vToken, a lightweight token-level virtualization layer that addresses the memory bottleneck in large language model serving. Existing KV cache eviction algorithms operate at a token granularity finer than block-level management, causing intra-block fragmentation and leaving a large fraction of allocated KV memory unreclaimable. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37\times. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2\times, while reducing the per-policy integration footprint from 500+ lines to under 50. Blogger's Review: vToken provides an effective solution to the KV cache eviction problem, its token-level virtualization technique can significantly reduce memory fragmentation, increase system throughput and concurrency, with a wide range of application prospects.

Original Source: https://arxiv.org/abs/2608.13263

[h] Back to Home