FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
This work addresses communication and computational bottlenecks in federated learning for clients with limited resources, representing an incremental improvement over existing methods.
The paper tackles the challenge of deploying large-scale models in federated learning with resource-constrained clients by proposing FedMoE-DA, a framework that uses a Mixture of Experts architecture and a domain-aware aggregation strategy to enhance robustness, personalization, and communication efficiency, achieving excellent performance while reducing server communication pressure.
Federated learning (FL) is a collaborative machine learning approach that enables multiple clients to train models without sharing their private data. With the rise of deep learning, large-scale models have garnered significant attention due to their exceptional performance. However, a key challenge in FL is the limitation imposed by clients with constrained computational and communication resources, which hampers the deployment of these large models. The Mixture of Experts (MoE) architecture addresses this challenge with its sparse activation property, which reduces computational workload and communication demands during inference and updates. Additionally, MoE facilitates better personalization by allowing each expert to specialize in different subsets of the data distribution. To alleviate the communication burdens between the server and clients, we propose FedMoE-DA, a new FL model training framework that leverages the MoE architecture and incorporates a novel domain-aware, fine-grained aggregation strategy to enhance the robustness, personalizability, and communication efficiency simultaneously. Specifically, the correlation between both intra-client expert models and inter-client data heterogeneity is exploited. Moreover, we utilize peer-to-peer (P2P) communication between clients for selective expert model synchronization, thus significantly reducing the server-client transmissions. Experiments demonstrate that our FedMoE-DA achieves excellent performance while reducing the communication pressure on the server.