Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce
This addresses bandwidth reduction for distributed deep learning training on clusters, but it is incremental as it builds on existing gradient compression techniques.
The paper tackles the problem of high bandwidth usage in large-scale distributed training by proposing an importance weighted pruning method for gradient compression on ring all-reduce, achieving compression ratios of 64X on AlexNet and 58.8X on ResNet50 on ImageNet while preserving accuracy.
It is inevitable to train large deep learning models on a large-scale cluster equipped with accelerators system. Deep gradient compression would highly increase the bandwidth utilization and speed up the training process but hard to implement on ring structure. In this paper, we find that redundant gradient and gradient staleness has negative effect on training. We have observed that in different epoch and different steps, the neural networks focus on updating different layers and different parameters. In order to save more communication bandwidth and preserve the accuracy on ring structure, which break the restrict as the node increase, we propose a new algorithm to measure the importance of gradients on large-scale cluster implementing ring all-reduce based on the size of the ratio of parameter calculation gradient to parameter value. Our importance weighted pruning approach achieved 64X and 58.8X of gradient compression ratio on AlexNet and ResNet50 on ImageNet. Meanwhile, in order to maintain the sparseness of the gradient propagation, we randomly broadcast the index of important gradients on each node. While the remaining nodes are ready for the index gradient and perform all-reduce update. This would speed up the convergence of the model and preserve the training accuracy.