Rate distortion comparison of a few gradient quantizers
This work addresses communication bottlenecks in distributed ML training, but it is incremental as it builds on existing quantization methods.
The paper analyzed the rate-distortion trade-offs of gradient quantization schemes like Scaled-sign and Top-K under a Gaussian assumption, comparing them to the Shannon limit and vector quantizers, with results showing specific performance gaps.
This article is in the context of gradient compression. Gradient compression is a popular technique for mitigating the communication bottleneck observed when training large machine learning models in a distributed manner using gradient-based methods such as stochastic gradient descent. In this article, assuming a Gaussian distribution for the components in gradient, we find the rate distortion trade-off of gradient quantization schemes such as Scaled-sign and Top-K, and compare with the Shannon rate distortion limit. A similar comparison with vector quantizers also is presented.