Parameter-efficient fine-tuning (PEFT) lets large language models quickly adapt to specialized tasks, yet it often harms the general abilities learned during pretraining. Existing solutions mainly rely on data replay or regularization, which require extra data or explicit constraints. We instead ask a more fundamental question: where in the model should adaptation be applied? Experiments reveal that fine‑tuning different Transformer layers yields varying gains on the target task and different degrees of capability loss, indicating that not all layers are equally suitable for adaptation. To quantify this, we compute layer‑wise empirical Fisher information as a sensitivity measure, but Fisher calculation needs backward passes and becomes costly for large models. We therefore introduce input‑output cosine similarity as a lightweight forward‑only proxy for ranking layer sensitivity. Across models and tasks, layers with lower similarity consistently show higher Fisher scores. Building on this insight, we propose Layer‑Selective LoRA (LS‑LoRA), which inserts trainable LoRA adapters only into layers with low input‑output similarity. Experiments on mathematical reasoning and code generation demonstrate that LS‑LoRA improves average target‑task performance while preserving much more commonsense reasoning ability than standard all‑layer LoRA. The results suggest that carefully choosing where to adapt offers a simple yet effective way to balance task adaptation and general capability retention.
Review