Head‑dimension compression can cut KV‑cache memory, yet most existing methods reconstruct full‑dimensional representations during inference, incurring large decoding FLOPs. We formalize the design considerations for converting pretrained attention into a natively reduced‑dimensional form that trims both KV‑cache storage and attention computation using a calibration set. The reductions satisfy three criteria: (1) a shared projection is applied to queries and keys so they stay in a common reduced feature space; (2) the projection is derived jointly from query and key activations to better preserve their interaction; (3) it remains compatible with the pretrained RoPE transformation. To achieve reconstruction‑free compression we restrict the projection to coordinate selection and prove that RoPE compatibility holds iff complete rotary pairs are retained. Building on this insight, RACE selects rotary pairs based on their joint query‑key activation energy. We theoretically show that this criterion minimizes a separable, calibration‑dependent upper bound on post‑RoPE query‑key product approximation error. The resulting reductions are folded into model weights, requiring no runtime reconstruction or specialized kernels. Across three model families and various compression levels, RACE outperforms structured pruning and stays competitive with reconstruction‑based methods on language modeling, zero‑shot, and long‑context tasks. At 50% KV‑cache retention and 256K context, RACE cuts total inference FLOPs by 47.8%, yields a 1.79× throughput boost, reduces time‑to‑first‑token by 43.2%, and lowers allocated GPU memory by 28.1%.
Review: RACE delivers substantial FLOPs and memory savings while preserving performance, offering a practical compression route that avoids extra runtime engineering.