In the field of computer vision, pre-trained Vision Foundation Models (VFMs) provide strong visual representations for various downstream tasks. However, the main challenge of VFM adaptation lies in the prohibitive costs of full fine-tuning and catastrophic forgetting. To address this, Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for Parameter-Efficient Fine-Tuning (PEFT).
Traditional LoRA is typically designed for transformer self-attention layers parameterized by 2D matrices. Since convolutional kernels inherently couple spatial and channel information within a 4D tensor, forcing them into a monolithic 2D matrix disrupts the inherent spatial topology. Therefore, this paper proposes Low-Rank Convolutional Adaptation (LoCA), a framework that addresses spatial-channel entanglement by decoupling channel and spatial adaptation.
LoCA introduces a low-rank channel adaptation mechanism for dense cross-channel mixing and refines spatial bases extracted from pre-trained kernels via Singular Value Decomposition (SVD). Experimental results show that LoCA preserves pre-trained spatial priors and achieves competitive or state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.
Blogger's Review: The introduction of LoCA presents a new approach for fine-tuning convolutional neural networks by decoupling spatial and channel adaptations, allowing for better utilization of pre-trained model advantages. Its promising performance in fine-grained classification and semantic segmentation tasks suggests a bright future for its applications in handling complex structured visual data.