Low-Rank Adaptation is widely used for parameter-efficient fine-tuning; however, existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements. This work theoretically and empirically demonstrates that uniform rank allocation is fundamentally suboptimal.
Motivated by this observation, we propose LAARA (Layer Aware Adaptive Rank Allocation framework), a search-free framework that dynamically allocates ranks using lightweight diagonal Fisher estimates computed during training.
LAARA combines projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism to produce stable and efficient rank adaptation. Experiments on the GLUE and MathInstruct benchmarks show that LAARA consistently matches or outperforms popular state-of-the-art approaches such as LoRA, AdaLoRA, DyLoRA, and Bitfit while using significantly fewer trainable parameters.
Our results indicate that Fisher-guided rank allocation provides a principled and effective foundation for adaptive parameter-efficient fine-tuning.
The code is publicly available at: LAARA.py
Blogger's Review: LAARA achieves remarkable parameter efficiency through dynamic rank allocation, addressing the shortcomings of traditional methods. This innovation highlights the importance of layer-aware mechanisms in fine-tuning and offers new insights for future model training that are worth exploring and applying.