The key‑value (KV) cache of autoregressive Transformers grows linearly with context length and dominates memory usage in long‑context scenarios. Most training‑free approaches prune low‑importance tokens, an irreversible operation along the sequence axis. AttSVD keeps every token and stores it more cheaply along the feature axis.
The core of AttSVD is an online, per‑prompt truncated SVD whose basis is derived from the prompt’s own attention geometry. It retains only the directions actually read by attention, reducing persistent per‑head KV memory proportionally to the retained rank. We introduce two decode‑time caching strategies:
- Accumulating for short‑generation regimes, where the full KV cache is kept and updated each step;
- Streaming for long‑generation regimes, where KV entries are refreshed and old ones released on the fly.
To make compression adaptive, AttSVD adds two refinements:
- Per‑matrix energy rule independently sizes the logit space and the attention mass, deciding the rank for each;
- Attention‑aware basis truncation cuts only in the subspaces that attention actually reads, preserving both attention logits and output.
These factors also provide free, per‑head interpretability, revealing the effective rank and the geometry consumed by attention. Across multiple models, on both an agentic benchmark and the full LongBench suite, AttSVD matches dense‑cache performance while using up to 50% of the KV‑cache memory.
Review: AttSVD leverages attention‑guided low‑rank decomposition to achieve substantial KV cache compression without losing token information, offering a practical and interpretable solution for long‑context inference.