Unmasking Efficiency: Learning Salient Sparse Models in Non-IID Federated Learning
This work addresses communication bottlenecks for federated learning systems, particularly in non-IID data scenarios, but it is incremental as it builds on existing sparse and federated learning methods.
The paper tackles the problem of communication inefficiency in non-IID federated learning by proposing Salient Sparse Federated Learning (SSFL), which identifies a sparse subnetwork before training using local saliency scores and aggregates them into a global mask, resulting in improved sparsity-accuracy trade-offs and reduced communication time in real-world deployment.
In this work, we propose Salient Sparse Federated Learning (SSFL), a streamlined approach for sparse federated learning with efficient communication. SSFL identifies a sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are communicated each round between the clients and the server. We validate SSFL's effectiveness using standard non-IID benchmarks, noting marked improvements in the sparsity--accuracy trade-offs. Finally, we deploy our method in a real-world federated learning framework and report improvement in communication time.