As the parameter size of language models continues to grow, effective model compression becomes increasingly important to reduce their computational and memory overhead. Existing compression methods suffer from bottleneck issues: as the compression ratio increases, performance degrades significantly. Low-rank decomposition and quantization are two prominent compression techniques that have been shown to significantly reduce the computational and memory requirements of Large Language Models (LLMs) while maintaining model accuracy. Evidently, combining these two methods could break through the existing compression bottleneck. However, how these two methods interact when combined remains a critical question for developers, as many assume they are orthogonal, meaning their combination would not introduce additional errors beyond those independently introduced by each method. This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal. We validate these findings through a series of experiments on large language models. Our results demonstrate that these methods are non-orthogonal, and their combination leads to significant performance degradation. Importantly, we propose a novel approach called the Diagonal Adhesive Method (DAM), which can effectively combine the two methods and mitigate the performance loss. Our research provides deep insights into model compression and lays a solid theoretical and experimental foundation for future related studies.
Blogger's Review: This research provides crucial theoretical support for advancements in compression technology by revealing the non-orthogonality of low-rank decomposition and quantization, driving further exploration in the field of model compression. The introduction of the Diagonal Adhesive Method offers new insights for practical applications, making it worthy of attention and further investigation.