Sparse tuning is widely adopted for adapting large language models because of its parameter efficiency. Its effectiveness depends not only on the number of trainable parameters but also on the feature‑interaction topology of the sparse update’s structural support. Limited or uneven connectivity can isolate subsets of feature dimensions, creating information islands that block direct task‑specific interactions. To address this, we propose Small‑World Induced Fine‑Tuning (SWIFT), a sparse tuning framework inspired by small‑world connectivity. SWIFT combines block‑diagonal sparse updates for local feature mixing with a fixed deterministic scattering permutation that routes the mixed outputs across feature groups without adding trainable parameters. Theoretically, we show that certain random sparse‑update models and block‑diagonal updates can produce information islands, whereas a sufficiently scattering permutation guarantees positive block‑level expansion when the trainable block‑diagonal factor consists of equal‑sized, structurally dense square blocks. Experiments on commonsense reasoning, natural language generation, and image classification demonstrate that SWIFT outperforms all evaluated parameter‑efficient fine‑tuning (PEFT) baselines on average and can match or surpass full fine‑tuning. The implementation is open‑source.
Review: SWIFT achieves global information flow with a lightweight permutation, delivering strong downstream performance while preserving parameter efficiency.