Equitable-FL: Federated Learning with Sparsity for Resource-Constrained Environment
This addresses the challenge of making federated learning accessible for devices with limited space, computing, or bandwidth, though it is incremental as it builds on existing sparsity techniques.
The paper tackles the problem of federated learning in resource-constrained environments by proposing a sparse method based on the Lottery Ticket Hypothesis to reduce model parameters without affecting accuracy, enabling participation from nodes with limited resources. Results on benchmark datasets like MNIST, CIFAR-10, and domain-specific data show efficacy in heterogeneous settings, with improvements in model size compaction and training speed-up.
In Federated Learning, model training is performed across multiple computing devices, where only parameters are shared with a common central server without exchanging their data instances. This strategy assumes abundance of resources on individual clients and utilizes these resources to build a richer model as user's models. However, when the assumption of the abundance of resources is violated, learning may not be possible as some nodes may not be able to participate in the process. In this paper, we propose a sparse form of federated learning that performs well in a Resource Constrained Environment. Our goal is to make learning possible, regardless of a node's space, computing, or bandwidth scarcity. The method is based on the observation that model size viz a viz available resources defines resource scarcity, which entails that reduction of the number of parameters without affecting accuracy is key to model training in a resource-constrained environment. In this work, the Lottery Ticket Hypothesis approach is utilized to progressively sparsify models to encourage nodes with resource scarcity to participate in collaborative training. We validate Equitable-FL on the $MNIST$, $F-MNIST$, and $CIFAR-10$ benchmark datasets, as well as the $Brain-MRI$ data and the $PlantVillage$ datasets. Further, we examine the effect of sparsity on performance, model size compaction, and speed-up for training. Results obtained from experiments performed for training convolutional neural networks validate the efficacy of Equitable-FL in heterogeneous resource-constrained learning environment.