NeFut Logo NeFut
中 Admin Login

[CS.AI] Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

Published at: 2026-10-13 22:00 Last updated: 2026-10-14 02:30
#AI #Machine Learning #LLM

Autoregressive generation suffers from a KV cache whose memory grows linearly with sequence length, batch size, and model depth. Existing solutions such as Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA) and Tensor Product Attention (TPA) reduce this cost but often trade off expressivity, require extra positional‑encoding machinery, or enforce restrictive low‑rank structures.\ \ We introduce Kronecker Product Attention (KPA), which represents keys and values as sums of Kronecker products of compact, token‑dependent matrix factors: $$K_i = \sum_{h=1}^{H} A_{i,h} \otimes B_{i,h}, \quad V_i = \sum_{h=1}^{H} C_{i,h} \otimes D_{i,h}$$ where $\otimes$ denotes the Kronecker product and $H$ is a tunable head‑split hyperparameter. KPA strictly generalizes TPA: the rank‑one vector outer‑product factorization appears as a degenerate Kronecker split, while non‑degenerate matrix factors capture richer block‑structured patterns.\ \ To preserve the relative‑position property of RoPE, we devise a structured rotary embedding operator that rotates only the compact factors, eliminating the need for a separate RoPE branch as required by MLA.\ \ Empirically, under identical training conditions across four model scales and nine downstream benchmarks, KPA matches or exceeds TPA, MHA, MQA, GQA, and MLA in accuracy while achieving up to 10× KV cache compression. The Kronecker head split is a design‑time hyperparameter that lets practitioners navigate the compression‑accuracy trade‑off without architectural changes.\ \ Review: By leveraging matrix‑level Kronecker factorization, KPA offers stronger representational power than vector outer‑product methods, maintains RoPE’s relative positioning, and dramatically cuts memory usage, making it a compelling addition to the KV compression toolbox.

Original Source: https://arxiv.org/abs/2610.12657

[h] Back to Home