Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
This work solves the problem of efficient large-scale distributed training for deep learning practitioners, though it appears incremental as it builds on existing methods with optimizations.
The paper tackled the challenges of scaling distributed deep learning to massive GPU clusters by addressing instability in large mini-batch training and gradient synchronization overhead, achieving ImageNet/ResNet-50 training in 122 seconds without significant accuracy loss.
Scaling the distributed deep learning to a massive GPU cluster level is challenging due to the instability of the large mini-batch training and the overhead of the gradient synchronization. We address the instability of the large mini-batch training with batch-size control and label smoothing. We address the overhead of the gradient synchronization with 2D-Torus all-reduce. Specifically, 2D-Torus all-reduce arranges GPUs in a logical 2D grid and performs a series of collective operation in different orientations. These two techniques are implemented with Neural Network Libraries (NNL). We have successfully trained ImageNet/ResNet-50 in 122 seconds without significant accuracy loss on ABCI cluster.