Weight-only post‑training quantization (PTQ) relies on minimizing reconstruction loss to preserve model quality at low bit‑widths. We show that weights obtained by minimizing this loss do not necessarily improve performance on new tasks, and can even degrade accuracy on the same calibration data. Our analysis reveals that when the distribution of input activations shifts, weights with lower reconstruction loss on the calibration set may incur higher loss under the new distribution. Motivated by these findings, we introduce Distributionally Robust Quantization (DRQ), which minimizes the worst‑case reconstruction loss over a constrained set of activation distributions. DRQ refines the integer codes of quantized weights within the existing quantization grid, leaving quantization parameters and inference operators unchanged, thus incurring no extra inference overhead. Extensive experiments demonstrate that DRQ consistently improves six representative PTQ methods—including AWQ, GPTQ, and ParoQuant—and yields gains on both dense and mixture‑of‑experts large language models. These results establish DRQ as a general post‑hoc refinement framework that enhances downstream performance without additional inference cost.
Review