FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE
This addresses the challenge of efficient and effective federated learning for large language models in resource-constrained environments, representing an incremental improvement over prior methods.
The paper tackles the problem of suboptimal performance in resource-adaptive federated fine-tuning of large language models due to compression, proposing FLAME, a framework based on Sparse Mixture-of-Experts that adapts by varying activated experts per client, which consistently outperforms existing methods across diverse computational settings.
Existing resource-adaptive LoRA federated fine-tuning methods enable clients to fine-tune models using compressed versions of global LoRA matrices, in order to accommodate various compute resources across clients. This compression requirement will lead to suboptimal performance due to information loss. To address this, we propose FLAME, a novel federated learning framework based on the Sparse Mixture-of-Experts (SMoE) architecture. Unlike prior approaches, FLAME retains full (uncompressed) global LoRA matrices and achieves client-side adaptability by varying the number of activated experts per client. However, incorporating SMoE into federated learning introduces unique challenges, specifically, the mismatch in output magnitude from partial expert activation and the imbalance in expert training quality across clients. FLAME tackles these challenges through a lightweight rescaling mechanism and an activation-aware aggregation scheme. Empirical results across diverse computational settings demonstrate that FLAME consistently outperforms existing methods, providing a robust and effective solution for resource-adaptive federated learning.