Parameter-Efficient Fine-Tuning (PEFT) has become a common approach for adapting large pre‑trained models. Existing reparameterization methods mainly rely on low‑rank or sparse structures. Low‑rank techniques reduce the number of parameters by constraining the rank of the update matrix, yet full fine‑tuning is inherently high‑rank, causing low‑rank methods to lag behind full fine‑tuning. Sparse methods prune weights to compress the model but restrict the flexibility of the parameter space and break connectivity among weights.
Toeplitz matrices have constant entries along each diagonal, offering a compact parameterization without imposing explicit rank or sparsity constraints. A straightforward idea is to update weights via a product of Toeplitz matrices, but, similar to recurrent neural networks, long multiplicative chains often suffer from gradient instability. To address this, the paper introduces ToPA (Block-wise Toeplitz Adaptation), which replaces standard Toeplitz matrices with block‑wise Toeplitz matrices. The block structure greatly increases the capacity of each factor and shortens the chain length, thereby stabilizing training and enhancing expressiveness.
Theoretically, products of block‑wise Toeplitz matrices can approximate arbitrary matrices with a shorter chain, providing a mathematical justification for ToPA’s design. Further analysis shows that ToPA offers greater expressivity than existing low‑rank and sparse parameterizations.
Empirical evaluations on 21 NLP and CV datasets across five model architectures consistently demonstrate that ToPA achieves significant performance gains while remaining parameter‑efficient.
Review