Federated Mixture of Experts
This addresses the problem of suboptimal performance in federated learning for users with heterogeneous data, though it appears incremental as it builds on existing mixture-of-experts and federated learning concepts.
The paper tackles data heterogeneity in federated learning by proposing Federated Mixture of Experts (FedMix), a framework that trains an ensemble of specialized models, and shows it improves performance compared to a single global model across various non-i.i.d. data sources.
Federated learning (FL) has emerged as the predominant approach for collaborative training of neural network models across multiple users, without the need to gather the data at a central location. One of the important challenges in this setting is data heterogeneity, i.e. different users have different data characteristics. For this reason, training and using a single global model might be suboptimal when considering the performance of each of the individual user's data. In this work, we tackle this problem via Federated Mixture of Experts, FedMix, a framework that allows us to train an ensemble of specialized models. FedMix adaptively selects and trains a user-specific selection of the ensemble members. We show that users with similar data characteristics select the same members and therefore share statistical strength while mitigating the effect of non-i.i.d data. Empirically, we show through an extensive experimental evaluation that FedMix improves performance compared to using a single global model across a variety of different sources of non-i.i.d.-ness.