CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
This work addresses communication overhead in distributed machine learning, which is a critical bottleneck for scaling training, but it appears incremental as it builds on existing compression and pipeline techniques.
The paper tackles the dual problems of increased computation cost and reduced convergence accuracy in distributed training when using gradient compression combined with pipeline parallelism, proposing a method to compensate for these delays and accuracy losses.
Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combining with parallel communication mechanism method like pipeline, gradient compression technique can greatly alleviate the impact of communication overhead. However, there exists two problems of gradient compression technique to be solved. Firstly, gradient compression brings in extra computation cost, which will delay the next training iteration. Secondly, gradient compression usually leads to the decrease of convergence accuracy.