Knowledge Distillation (KD) has gained significant attention in Large Language Models (LLMs), yet the underlying mechanisms of its effectiveness remain unclear. This paper proposes a unified approach to explore the common mechanisms behind various KD methods through interactions.
Specifically, it decomposes the output score of the LLM into the sum of numerous interactions, where each interaction represents a nonlinear relationship involving a set of input variables (e.g., words). Based on these decomposed interactions, the study reveals that the common mechanism among different KD methods is the sparsification of interactions, meaning that student models retain fewer interactions during inference while suppressing others to zero effect.
Furthermore, it discovers that performance variance across different KD methods arises from their ability to handle complex interactions. A KD method typically performs better if it allows the student model to achieve higher sparsity of complex interactions. Motivated by these insights, the authors propose a plug-and-play loss function called Complex Interaction Penalty (CIP) to explicitly enforce the sparsity of complex interactions during the distillation process. Extensive experiments demonstrate that integrating CIP consistently enhances the performance of various KD methods on both in-domain and out-of-distribution benchmarks.
Blogger's Review: This research delves into the intrinsic mechanisms of knowledge distillation, offering new perspectives on the training processes of large language models. By introducing the Complex Interaction Penalty loss function, the authors not only pave the way for performance improvements but also guide future research directions. Effective interaction sparsification is poised to become a vital strategy for enhancing models.