Federated fine‑tuning of large language models faces heterogeneous client capacities, and low‑rank adaptation (LoRA) with client‑specific ranks $r_i$ makes deployment feasible. Existing server‑side aggregators first compose local LoRA factors $(A_i,B_i)$ into matrix updates $\Delta W_i = A_i B_i^{\top}$ and then average them directly in the weight space $\mathbb{R}^{m\times n}$. Although the updates share the same space, the factors are expressed in client‑specific local bases, causing basis inconsistency. In the usual compose‑aggregate‑factorize‑dispatch loop, this inconsistency can render the rank‑$r_i$ prefix dispatched to lower‑rank clients unstable after server‑side refactorization.
To address this, we propose FedAbA (Basis‑Consistent Federated LoRA Aggregation). The server first extracts each client’s local basis and aligns it to the corresponding prefix of a global reference basis. Using the aligned basis, it reconstructs each $\Delta W_i$ into a basis‑consistent matrix. These reconstructions are aggregated with weights that combine client data size and a consistency score. After aggregation, an exact SVD refactorization is performed in coefficient space, followed by gauge fixing, and the rank‑prefix is safely dispatched to clients.
Our theoretical analysis shows that basis consistency reduces aggregation error and justifies the consistency‑aware weighting scheme. Extensive experiments across various heterogeneous‑rank scenarios demonstrate that FedAbA consistently outperforms representative federated LoRA baselines, with the largest gains observed under the strongest rank heterogeneity.
Review: FedAbA eliminates hidden bias caused by mismatched bases, offering a robust and efficient framework for heterogeneous LoRA federated learning.