Abstract
Large language models (LLMs) demonstrate strong performance across diverse tasks, yet their deployment is limited by the memory and computational costs of their parameters. Structured pruning addresses this by removing entire structures, such as attention heads and Multi-Layer Perceptron (MLP) neurons, to create smaller dense models that operate efficiently on standard hardware. However, current methods rely on either gradient-based importance estimation, which is memory-intensive, or activation-based statistical proxies that do not directly measure the impact of removal on the loss. Moreover, the interaction between the importance criterion and the post-pruning recovery strategy has not been systematically examined.
We introduce TriSP (Tri-Signal Structured Pruning), an importance metric that combines weight magnitude scaled by activation norm with first-order gradient sensitivity via a geometric mean, yielding a channel-level score that captures both structural and loss-sensitivity signals. Coupled with adaptive per-layer budget allocation and low-rank adaptation (LoRA) recovery, TriSP achieves the lowest perplexity and highest zero-shot accuracy across all tested configurations, attaining a perplexity of 6.80 on WikiText-2 at 20% pruning on LLaMA-7B. Inference throughput improves by 82% at 50% pruning while maintaining competitive performance.
Blogger's Review: The TriSP method significantly enhances the efficiency of pruning in large language models by integrating weight and activation signals, providing new possibilities for model utility in resource-constrained environments. Its innovative geometric mean strategy ensures performance retention during pruning, making this approach highly valuable in practical applications.