ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning
This work addresses stability challenges in decentralized federated learning for fine-tuning large models, offering an incremental improvement over existing LoRA methods.
This paper tackled the problem of stabilizing low-rank updates in decentralized federated fine-tuning by introducing ADF-LoRA, which synchronizes updates of one low-rank matrix per round and mixes matrices to improve consistency. The result showed faster and smoother convergence, achieving the highest average accuracy across GLUE tasks and outperforming existing LoRA variants by a consistent margin.
This paper revisits alternating low-rank updates for federated fine-tuning and examines their behavior in decentralized federated learning (DFL). While alternating the LoRA matrices has been shown to stabilize aggregation in centralized FL, extending this mechanism to decentralized, peer-to-peer communication introduces new challenges due to phase-state mismatch and block-wise divergence across clients. We introduce ADF-LoRA, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation. This design preserves the cross-term suppression effect of alternating updates while improving stability in serverless topologies. We provide a convergence analysis under standard smoothness assumptions and evaluate ADF-LoRA on multiple GLUE tasks. Experiments show that ADF-LoRA achieves faster and smoother convergence and delivers the highest average accuracy across tasks, outperforming existing LoRA variants in decentralized FL by a consistent margin.