Subspace-Constrained Federated Learning with Low-Rank Adaptation
For practitioners of federated fine-tuning of large models, this work provides a method to mitigate destructive aggregation from heterogeneous client data, but the improvement is model-dependent and incremental over existing federated LoRA approaches.
The paper addresses geometric misalignment in federated low-rank adaptation (LoRA) and proposes a subspace-regularized objective that keeps local updates close to a shared global subspace. On RoBERTa-large, the method achieves best mean accuracy of 0.454 ± 0.023 and near-perfect basis overlap (~0.9999), outperforming baselines, though gains are model-dependent (FedAvg leads on SmolLM-360M).
Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints, but heterogeneous client data can induce geometric misalignment between local low-rank updates. We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning. We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace. We present a complete empirical evaluation on two pretrained models, RoBERTa-large and SmolLM-360M, over HellaSwag in a non-IID 10-client federated setting, across 3 random seeds (42, 43, 44), yielding 24 total experimental runs (4 methods x 3 seeds x 2 models). On RoBERTa-large, Subspace-Reg achieves the strongest mean best accuracy (0.454 +/- 0.023), mean final accuracy (0.429 +/- 0.011), and lowest final loss (1.363) across all three seeds, outperforming FedAvg, SVD redistribution, and FedSVD baselines by a large margin. On SmolLM-360M, FedAvg leads on accuracy, revealing that accuracy gains are model-dependent. Crucially, Subspace-Reg achieves near-perfect basis overlap, approximately 0.9999, on both models and across all seeds, versus 0.958 to 0.991 for all baselines, providing robust support for the geometric alignment hypothesis. The code is publicly available at https://github.com/sadia-sigma-lab/Subspace-Constrained-Federated-learning-with-Lora.