Multi‑head latent attention (MLA) exposes many logical query heads through a single packed latent KV stream. This representation is memory‑efficient but removes the physical per‑head cache boundary assumed by conventional head‑wise reuse. We implement RedKnot’s head‑aware reuse principle on DeepSeek‑V4. Each immutable document is processed offline at canonical position zero; certified Local‑head contributions are retained as MLA‑Off. At serving time, query‑side RoPE relocation restores the document’s request position, a small set of Global‑heads and protected Local token rows are recomputed as MLA‑Online, and the two paths are merged before a shared output projection, never splitting the packed MLA latent. DeepSeek‑V4‑Flash uses 37 reusable layers with a 56/8 Local/Global head partition, achieving a 75.29% analytic logical head‑row ceiling; the Pro‑0813 profile uses 55 layers and 112/16 heads, reaching 78.89%. Frozen Flash operating points show hot‑artifact TTFT speedups of 2.02‑3.84×. In a 256K three‑dataset archived study, the aggregate F1 improves by 3.24 points, EM by 4.16 points, and analytic major‑operator arithmetic saving is 78.7‑79.5%, while one dataset drops by 2.81 F1 points. An author‑reported 256K hot‑artifact QPS measurement is roughly 2.0×; because the raw concurrency trace is not included, we label it as preliminary rather than archived evidence. The paper also details factorization, position repair, token‑row closure, sparse‑MoE support, TP8 integration, and the measurement boundaries needed to interpret these results.
Review