Learnings from Federated Learning in the Real world
This addresses data heterogeneity issues in real-world Federated Learning for NLU systems, but it is incremental as it builds on existing FL methods with a sampling tweak.
The study tackled the problem of data distribution idiosyncrasies, such as heavy devices with large data and light users with little data, in Federated Learning for Natural Language Understanding, showing that simple non-uniform device selection based on interactions boosts model performance and catches up with methods using all data in continual FL.
Federated Learning (FL) applied to real world data may suffer from several idiosyncrasies. One such idiosyncrasy is the data distribution across devices. Data across devices could be distributed such that there are some "heavy devices" with large amounts of data while there are many "light users" with only a handful of data points. There also exists heterogeneity of data across devices. In this study, we evaluate the impact of such idiosyncrasies on Natural Language Understanding (NLU) models trained using FL. We conduct experiments on data obtained from a large scale NLU system serving thousands of devices and show that simple non-uniform device selection based on the number of interactions at each round of FL training boosts the performance of the model. This benefit is further amplified in continual FL on consecutive time periods, where non-uniform sampling manages to swiftly catch up with FL methods using all data at once.