On the Utility of Gradient Compression in Distributed Training Systems
This work addresses communication bottlenecks in distributed training systems, but it is incremental as it critiques and refines existing compression methods rather than introducing new ones.
The paper evaluated gradient compression methods for distributed training, finding that they provided speedup in only 6 out of over 200 setups compared to optimized synchronous SGD, and identified root causes and desirable properties for effective compression.
A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, a long line of recent work proposes gradient and model compression methods. In this work, we evaluate the efficacy of gradient compression methods and compare their scalability with optimized implementations of synchronous data-parallel SGD across more than 200 different setups. Surprisingly, we observe that only in 6 cases out of more than 200, gradient compression methods provide speedup over optimized synchronous data-parallel training in the typical data-center setting. We conduct an extensive investigation to identify the root causes of this phenomenon, and offer a performance model that can be used to identify the benefits of gradient compression for a variety of system setups. Based on our analysis, we propose a list of desirable properties that gradient compression methods should satisfy, in order for them to provide a meaningful end-to-end speedup.