Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning
This addresses the challenge of scaling federated learning to large models for edge devices with constrained resources, representing a novel method rather than an incremental improvement.
The paper tackles the problem of training large models in federated learning despite clients' limited resources by proposing Fed-ET, a method where small heterogeneous models on clients train a larger server model via ensemble knowledge transfer, resulting in significant performance improvements over state-of-the-art FL algorithms with fewer communicated parameters and robustness to data heterogeneity.
Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL the ensemble can be trained on clients' highly heterogeneous data. Cognizant of this property, Fed-ET uses a weighted consensus distillation scheme with diversity regularization that efficiently extracts reliable consensus from the ensemble while improving generalization by exploiting the diversity within the ensemble. We show the generalization bound for the ensemble of weighted models trained on heterogeneous datasets that supports the intuition of Fed-ET. Our experiments on image and language tasks show that Fed-ET significantly outperforms other state-of-the-art FL algorithms with fewer communicated parameters, and is also robust against high data-heterogeneity.