Hybrid large language models (LLMs) combine softmax attention layers with linear‑attention modules such as Gated DeltaNet (GDN). GDN keeps a recurrent state that summarizes the context in a fixed‑size vector. Early community 4‑bit quantizations of Qwen3.8‑27B (48 GDN layers, 16 attention layers) kept the GDN block in 8‑ or 16‑bit precision—especially its decay and write‑strength gates—based on the intuition that recurrence errors would accumulate over long contexts.
We built Minima: NVFP4 W4A4, quantizing all 496 linear layers (including GDN) to $4$‑bit. Across perplexity at 4K/32K, MMLU‑Pro, GSM8K, AIME'25, GPQA‑Diamond, LiveCodeBench, and RULER retrieval (up to 64K), Minima matches BF16 within seed noise (5‑task average $-0.52$), while being the smallest model ($17.5\,\text{GiB}$) and the fastest‑prefill recipe (+14‑19%). Its 32K perplexity gap shrinks with token position.
A four‑part mechanism study explains why:
- NVFP4’s 16‑element block scaling localizes extreme outliers in the residual stream, equalizing activation error across layer roles;
- The supposedly fragile gate projections are actually the least sensitive—softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error;
- The delta‑rule recurrence holds injected noise at a flat plateau over ~32K tokens and forgets a state impulse within a few hundred steps, because each write overwrites the state along the current key direction;
- Per‑token quantization cost is washed out by context rather than compounding.
We also repaired a global‑scale mismatch that appears when per‑module‑calibrated NVFP4 checkpoints are served by kernels that fuse those modules into a single GEMM. Calibrated FP8 KV‑cache scales prove performance‑free.
Practical recipe: quantize everything, ship KV scales. The result is a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint is available at https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4
Review