Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
This work addresses the problem of efficiently fine-tuning large, sparsely-activated models in federated settings for participants with limited computing resources, representing an incremental improvement over prior approaches.
The paper tackles the challenge of federated fine-tuning of Mixture-of-Experts (MoE)-based large language models on resource-constrained devices by proposing FLUX, a system that achieves up to 4.75X speedup in time-to-accuracy compared to existing methods.
Federated fine-tuning of Mixture-of-Experts (MoE)-based large language models (LLMs) is challenging due to their massive computational requirements and the resource constraints of participants. Existing working attempts to fill this gap through model quantization, computation offloading, or expert pruning. However, they cannot achieve desired performance due to impractical system assumptions and a lack of consideration for MoE-specific characteristics. In this paper, we propose FLUX, a system designed to enable federated fine-tuning of MoE-based LLMs across participants with constrained computing resources (e.g., consumer-grade GPUs), aiming to minimize time-to-accuracy. FLUX introduces three key innovations: (1) quantization-based local profiling to estimate expert activation with minimal overhead, (2) adaptive layer-aware expert merging to reduce resource consumption while preserving accuracy, and (3) dynamic expert role assignment using an exploration-exploitation strategy to balance tuning and non-tuning experts. Extensive experiments on LLaMA-MoE and DeepSeek-MoE with multiple benchmark datasets demonstrate that FLUX significantly outperforms existing methods, achieving up to 4.75X speedup in time-to-accuracy.