ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
This addresses the bottleneck of distributed training efficiency for deep learning practitioners, though it is incremental as it builds on existing ResNet and distributed training methods.
The authors tackled the problem of high communication and computational costs in distributed training of Residual Networks by proposing ResIST, a protocol that decomposes ResNets into shallow sub-networks trained independently, reducing per-iteration communication, memory, and time requirements to a fraction of full-model training while maintaining competitive model performance.
We propose ResIST, a novel distributed training protocol for Residual Networks (ResNets). ResIST randomly decomposes a global ResNet into several shallow sub-ResNets that are trained independently in a distributed manner for several local iterations, before having their updates synchronized and aggregated into the global model. In the next round, new sub-ResNets are randomly generated and the process repeats until convergence. By construction, per iteration, ResIST communicates only a small portion of network parameters to each machine and never uses the full model during training. Thus, ResIST reduces the per-iteration communication, memory, and time requirements of ResNet training to only a fraction of the requirements of full-model training. In comparison to common protocols, like data-parallel training and data-parallel training with local SGD, ResIST yields a decrease in communication and compute requirements, while being competitive with respect to model performance.