Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods
This work addresses communication overheads in distributed training for deep learning practitioners, but it is incremental as it builds on existing local gradient methods like Local SGD.
The paper tackles the problem of communication bottlenecks in distributed training of deep neural networks by introducing adaptive batch size strategies for local gradient methods, which reduce minibatch gradient variance and improve training efficiency and generalization, as demonstrated in image classification and language modeling experiments.
Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the main bottleneck in data-parallel minibatch stochastic gradient methods with per-iteration gradient synchronization. Local gradient methods like Local SGD reduce communication by only synchronizing model parameters and/or gradients after several local steps. Despite an understanding of their convergence and the importance of batch sizes for training efficiency and generalization, optimal batch sizes for local gradient methods are difficult to determine. We introduce adaptive batch size strategies for local gradient methods that increase batch sizes adaptively to reduce minibatch gradient variance. We provide convergence guarantees under homogeneous data conditions and support our claims with image classification and language modeling experiments, demonstrating the effectiveness of our strategies for both training efficiency and generalization.