Revisiting Ensembling in One-Shot Federated Learning
This work addresses the trade-off between accuracy and communication efficiency in federated learning for distributed systems, offering a practical improvement over existing OFL methods.
The paper tackles the performance gap in one-shot federated learning (OFL) under data heterogeneity by introducing FENS, a federated ensembling scheme that achieves up to 26.9% higher accuracy than SOTA OFL on CIFAR-10, with only 3.1% lower accuracy than iterative FL and at most 4.3x more communication than OFL.
Federated learning (FL) is an appealing approach to training machine learning models without sharing raw data. However, standard FL algorithms are iterative and thus induce a significant communication cost. One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits a performance gap in terms of accuracy with respect to FL, especially under high data heterogeneity. We introduce FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL. Learning in FENS proceeds in two phases: first, clients train models locally and send them to the server, similar to OFL; second, clients collaboratively train a lightweight prediction aggregator model using FL. We showcase the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL. At the same time, FENS incurs at most 4.3x more communication than OFL, whereas FL is at least 10.9x more communication-intensive than FENS.