IntSGD: Adaptive Floatless Compression of Stochastic Gradients
This work addresses communication bottlenecks in distributed machine learning, offering a practical solution for reducing bandwidth usage, though it builds incrementally on prior integer compression methods.
The authors tackled the problem of communication overhead in distributed Stochastic Gradient Descent by proposing IntSGD, a family of adaptive integer compression operators that eliminate floating-point communication, achieving provable convergence with iteration complexity matching SGD up to constant factors for various function types.
We propose a family of adaptive integer compression operators for distributed Stochastic Gradient Descent (SGD) that do not communicate a single float. This is achieved by multiplying floating-point vectors with a number known to every device and then rounding to integers. In contrast to the prior work on integer compression for SwitchML by Sapio et al. (2021), our IntSGD method is provably convergent and computationally cheaper as it estimates the scaling of vectors adaptively. Our theory shows that the iteration complexity of IntSGD matches that of SGD up to constant factors for both convex and non-convex, smooth and non-smooth functions, with and without overparameterization. Moreover, our algorithm can also be tailored for the popular all-reduce primitive and shows promising empirical performance.