FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
This addresses scalability issues in federated learning for clients with limited resources, though it is incremental as it builds on split learning.
The paper tackles the high communication cost in split learning for federated learning on resource-constrained clients by introducing a clustering scheme and gradient correction method, achieving up to 490x communication reduction with minimal accuracy drop.
In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating server. However, updating and communicating the entire model becomes prohibitively expensive when resource-constrained clients collectively aim to train a large machine learning model. Split learning provides a natural solution in such a setting, where only a small part of the model is stored and trained on clients while the remaining large part of the model only stays at the servers. However, the model partitioning employed in split learning introduces a significant amount of communication cost. This paper addresses this issue by compressing the additional communication using a novel clustering scheme accompanied by a gradient correction method. Extensive empirical evaluations on image and text benchmarks show that the proposed method can achieve up to $490\times$ communication cost reduction with minimal drop in accuracy, and enables a desirable performance vs. communication trade-off.