Learning effectively from limited data is crucial in domains like security where labeled examples are scarce. Large language models (LLMs) have shown some capability for data-efficient learning, particularly through parameter-efficient adaptation methods, but they still face challenges with few samples for difficult tasks. To address this, we propose a method called Attention Head Reweighting (AHR), which adapts LLMs to new text classification tasks by learning a single scalar per attention head. This drastically reduces the number of parameters to be learned by leveraging the functional specialization of individual attention heads.
Experiments on various open-source text classification datasets demonstrate that AHR can outperform standard baselines like LoRA when learning from limited samples, despite having 200-1000x fewer trainable parameters, as AHR modifies only about 0.0001% of the model's parameters. Additionally, the learned weights are easy to interpret and can be analyzed to better understand the mechanisms and attention heads responsible for in-context learning abilities in LLMs.
Blogger's Review: The AHR method significantly enhances learning efficiency under data scarcity by simplifying the learning process, providing a new perspective and tools for the application of large language models, especially in fields like security. The interpretability of this method also facilitates model debugging and optimization, making it a valuable contribution to the field.