Post‑training quantization (PTQ) compresses a trained model’s weights with limited compute and data. When quantized to 4 bits the loss is negligible, but scalar PTQ at 2 bits drops accuracy to chance level. We examine three factors usually treated as implementation details: the quantization representation, the calibration data, and the recovery fine‑tuning data, across six dense and mixture‑of‑experts models ranging from $3.8$ to $32$ B parameters.
Representation is the dominant factor; vector codebooks can revive models where scalar PTQ collapses. The remaining accuracy gap appears mainly on generative tasks and varies by model. With a short recovery budget, calibration and recovery data become coupled – the optimal recovery set depends on the calibration set, so they must be chosen jointly.
Our recipe therefore uses a vector‑codebook representation, a small task‑aligned calibration mixture, and an inexpensive codebook‑only recovery stage. The approach consumes about $2.3$ effective bits per weight and retains $89\%$–$97\%$ of full‑precision question‑answering accuracy where scalar PTQ fails. Compared with the strongest quantization‑aware‑training (QAT) baseline, it requires $1.4$–$30\times$ less compute.
Review: The study shows that even at extreme 2‑bit precision, a data‑centric choice of representation and a joint calibration‑recovery strategy can preserve useful performance, offering a more efficient alternative to conventional QAT.