FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
This addresses the problem of enhancing model capacity and performance in federated learning for applications with privacy constraints, representing a novel paradigm rather than an incremental improvement.
The paper tackles the limitation of model complexity in federated learning due to resource-constrained edge devices by introducing a framework that uses a powerful server model to selectively learn from clients and transfer knowledge back, achieving superior performance on both server and client models in image classification tasks.
Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.