Improving Accuracy of Federated Learning in Non-IID Settings
This work addresses performance degradation in Federated Learning for privacy-sensitive applications, but it is incremental as it builds on existing methods with specific optimizations.
The paper tackled the problem of Federated Learning performing poorly in non-IID data settings by developing techniques to improve model accuracy without extra communication overhead, resulting in a validation accuracy increase of over 12% compared to the baseline.
Federated Learning (FL) is a decentralized machine learning protocol that allows a set of participating agents to collaboratively train a model without sharing their data. This makes FL particularly suitable for settings where data privacy is desired. However, it has been observed that the performance of FL is closely tied with the local data distributions of agents. Particularly, in settings where local data distributions vastly differ among agents, FL performs rather poorly with respect to the centralized training. To address this problem, we hypothesize the reasons behind the performance degradation, and develop some techniques to address these reasons accordingly. In this work, we identify four simple techniques that can improve the performance of trained models without incurring any additional communication overhead to FL, but rather, some light computation overhead either on the client, or the server-side. In our experimental analysis, combination of our techniques improved the validation accuracy of a model trained via FL by more than 12% with respect to our baseline. This is about 5% less than the accuracy of the model trained on centralized data.