Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
For researchers and practitioners in federated learning and LLM fine-tuning, this work addresses a critical bottleneck in integrating MoE with FL under data heterogeneity, offering a practical solution to improve model performance and training stability.
The paper tackles the problem of conflicting expert optimizations in federated fine-tuning of Mixture-of-Experts LLMs under data heterogeneity. The proposed FC-MoE framework uses importance-aware weighting, gradient consensus projection, and local knowledge retention to stabilize training, achieving faster convergence and improved global/local performance in non-IID settings.
The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation. While federated learning (FL) emerges as the paradigm for privacy-preserving collaborative optimization, integrating MoE into FL under data heterogeneity may trigger conflicting expert optimizations. Client-specific data distributions force same-indexed experts to optimize under inconsistent or even conflicting feature-label correlations. This mismatch induces destructive interference during aggregation, thus destabilizing the optimization trajectory and degrading model performance. To address this issue, we propose FC-MoE, a federated conflict-aware framework for MoE fine-tuning. It employs an importance aware weighting scheme to prioritize reliable local updates and utilizes gradient consensus projection to suppress conflicting updates, ensuring a stable global optimization path. Moreover, a local knowledge retention mechanism further preserves specialized client expertise by re-anchoring domain-specific residuals. Extensive experiments demonstrate that FC-MoE accelerates convergence and enhances both global and local model performance in non-IID federated environments.