Massive activations refer to the phenomenon where a few hidden channels in large language models (LLMs) exhibit unusually large magnitudes, and this pattern appears across many architectures. The authors identify a single input‑embedding channel that governs this behavior; its position is fixed for a given model and is named the Massive Activation Gating Channel (MAGC). When the MAGC value becomes sufficiently large (or sufficiently small, depending on the model), the subsequent spike feed‑forward network (FFN) produces massive activations.
Experiments on six LLMs from four model families of varying sizes confirm both the existence of MAGC and its effect. Theoretical analysis shows that, under extreme MAGC values, the spike FFN output asymptotically reduces to a quadratic form: $$ \mathbf{y} \approx \mathbf{x}^T \mathbf{W}_{down}^{(S)} \mathbf{x} $$ where $\mathbf{W}_{down}^{(S)}$ denotes a few columns of the FFN down‑projection matrix. These columns already display the massive‑activation shape, so the output inherits the same pattern.
The study uncovers a fine‑grained mechanism behind activation distribution in LLMs, offering insights for sparsity interpretation and potential operator optimizations.
Review