Communication-Efficient Distributed SGD with Compressed Sensing
This work addresses communication efficiency for edge devices in federated learning, though it is incremental as it builds on existing compressed sensing techniques.
The paper tackles the communication bottleneck in distributed SGD by proposing an algorithm that uses compressed sensing to exploit gradient sparsity, achieving a 30% reduction in communication cost with comparable convergence to standard methods.
We consider large scale distributed optimization over a set of edge devices connected to a central server, where the limited communication bandwidth between the server and edge devices imposes a significant bottleneck for the optimization procedure. Inspired by recent advances in federated learning, we propose a distributed stochastic gradient descent (SGD) type algorithm that exploits the sparsity of the gradient, when possible, to reduce communication burden. At the heart of the algorithm is to use compressed sensing techniques for the compression of the local stochastic gradients at the device side; and at the server side, a sparse approximation of the global stochastic gradient is recovered from the noisy aggregated compressed local gradients. We conduct theoretical analysis on the convergence of our algorithm in the presence of noise perturbation incurred by the communication channels, and also conduct numerical experiments to corroborate its effectiveness.