Abstract
Long-context LLM inference is increasingly constrained by the memory and bandwidth costs of KV caches, yet aggressive compression can compromise the layer-specific evidence required for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes based on their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structures without retraining.
Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under significantly smaller cache budgets. With a 32k-token prefill window, FreqDepthKV achieves 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also enhances decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.
Blogger's Review: The introduction of FreqDepthKV presents an innovative solution for cache management in long-context LLM inference. Its frequency-guided depth sharing mechanism effectively balances compression rates and inference performance, showcasing an elegant equilibrium between computational efficiency and accuracy. This method undoubtedly opens new avenues for future LLM applications, especially in resource-constrained environments.