GAAC (Group‑Adaptive Additive Codec) is a lossless KV‑cache compression and serving system designed for long‑context LLM inference on personal devices. It exploits the high redundancy of BF16 exponent bits by partitioning the exponent stream per channel and encoding it with a base‑plus‑offset scheme over contiguous ranges. The core steps are:
- For each channel compute the minimum base
baseand the maximum offsetoffset, then select the smallest bit‑widthwthat can represent the range. - Within a small group (typically 64‑256 KV entries) all channels share the same
w, achieving group adaptivity. - The resulting bit‑stream can be decoded block‑wise, avoiding a full‑cache decompression pass.
During attention, GAAC employs a fused kernel that reconstructs the BF16 values on‑the‑fly for the required KV tiles and feeds them directly into the attention matrix multiplication, eliminating the overhead of full cache expansion.
The system also provides block‑level compression and fixed‑capacity slots, enabling incremental cache updates and paged serving that respect GPU/CPU memory limits.
Benchmarks on five mainstream models show a lossless BF16 KV compression ratio of 1.405‑1.407×. For low‑precision caches the compression gains are:
- FP8 E4M3: 1.09‑1.14×
- FP8 E5M2: 1.25‑1.30×
On an NVIDIA DGX Spark, GAAC speeds up decode‑attention at a 64K‑token context by 1.19‑1.27× and improves end‑to‑end W4A16 decode steps by 1.04‑1.15×. In serving scenarios, resident KV memory drops by roughly 26% while aggregate throughput rises 13‑29% under comparable KV‑memory budgets. On an Apple M4, GAAC delivers 1.02‑1.25× attention speedups, demonstrating cross‑platform portability.
Review: GAAC’s fine‑grained encoding of exponent redundancy delivers both lossless compression and attention acceleration, making it a practical breakthrough for resource‑constrained personal devices and long‑context LLM inference.