Abstract
Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization.
We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts.
SOS-LoRA features:
- Decomposing the total rank across experts;
- Applying a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics;
- Promoting diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer.
SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at GitHub.
Blogger's Review: SOS-LoRA enhances the flexibility and performance of low-rank adaptation by introducing static experts and multi-scale optimization, particularly excelling in complex reasoning tasks. This provides a fresh perspective for future model fine-tuning, warranting deeper exploration and application.