Quantization and pruning are common techniques to reduce the deployment cost of large language models, yet their impact is usually reported only by benchmark scores. Such scores capture the residual loss at the output layer but ignore how different internal components are damaged and how the rest of the network compensates. The observed loss therefore mixes damage and compensation, and it is unclear whether their balance changes with model size.
This work focuses on in‑context retrieval, a capability carried by a small, identifiable set of attention heads. To disentangle damage from compensation, the expected loss caused by ablating these heads is exactly decomposed into two parts: the portion restored by the remaining network, attributed to specific compensating components, and the portion that reaches the output. The decomposition is then compared across models of varying scale.
The findings show that for comparable damage, smaller models recover most of the loss, whereas larger models retain most of it. In other words, as models grow, the compensating response of the network diminishes. Compressing the same heads yields the same pattern—compression that is almost harmless in small models can become costly in large ones.
These results suggest that compression techniques may be riskier for large models and should be re‑evaluated for cost‑effectiveness in real deployments.\n\nReview