Mixture‑of‑experts (MoE) architectures enable large language models to scale efficiently by routing inputs to specialized experts, but they also require massive GPU memory. Residual sparsification is a common compression technique that splits each expert's projection matrix into a shared base matrix and an expert‑specific residual matrix, then sparsifies the residuals. Existing approaches compress each residual independently by minimizing its reconstruction error, which effectively minimizes the error of each projection matrix. However, an expert’s final output results from coupled computations across multiple projections and hidden representations, so even tiny errors in individual matrices can propagate and amplify, causing large output errors and accuracy loss.
To align the compression objective with actual model performance, we propose PARSER. Instead of focusing on isolated matrix errors, PARSER preserves the expert output error by introducing output importance, a metric that quantifies each residual’s contribution to the final output error. The sparsification budget is allocated according to this importance, keeping critical residuals intact while aggressively compressing less influential ones.
Experiments on the Qwen and DeepSeek benchmarks show that, under the same peak‑memory reduction, PARSER narrows the accuracy gap to the uncompressed model by 1.41× on Qwen and 1.44× on DeepSeek, demonstrating a clear advantage over prior methods.
Review: PARSER’s output‑centric objective bridges the gap between compression and accuracy, offering a practical pathway for deploying large‑scale MoE models with limited memory.