Extreme low‑bit weight‑only quantization is a practical way to cut deployment costs of large language models. Scalar quantization below 3 bits often incurs severe accuracy loss, while vector quantization (VQ) preserves performance better by using richer joint encoding. Existing VQ approaches still degrade noticeably on real‑world tasks, mainly due to two under‑explored challenges: (1) protecting sensitive weight channels inside jointly encoded vectors; (2) allocating a fixed bit budget among representation components such as direction codebooks and radius precision for different weight matrices. To tackle these issues, we introduce SAGE (Split Adaptation and Grouping by Energy), a weight‑only VQ method. Energy‑Stratified Grouping (ESG) couples channel grouping with activation‑weighted fitting, mixing channels from different energy tiers within each sub‑vector to prioritize sensitive ones. Adaptive Bit Split (ABS) balances direction‑codebook capacity and radius precision per matrix under a fixed budget, selecting the split that yields the lowest activation‑weighted reconstruction error. Benchmarks spanning knowledge QA, math reasoning, code generation, instruction following and tool use show state‑of‑the‑art results in the 2‑bit setting. On Qwen3‑30B‑A3B, SAGE retains 95.3% of the BF16 model’s average score. With a custom CUDA kernel, SAGE achieves up to 2.55× non‑streaming end‑to‑end speedup over FP16. The implementation is open‑source at https://github.com/Zincwolf/sage.
Review: By combining energy‑aware grouping with per‑matrix bit allocation, SAGE mitigates the typical accuracy drop of ultra‑low‑bit VQ while delivering substantial speed gains, marking a significant step toward practical deployment of 2‑bit LLMs.