Locally Asynchronous Stochastic Gradient Descent for Decentralised Deep Learning
This addresses communication bottlenecks in decentralized deep learning, offering a practical improvement over existing methods.
The paper tackles the communication slowdowns in distributed deep learning by proposing Local Asynchronous SGD (LASGD), an asynchronous decentralized algorithm using All Reduce for synchronization, which accelerates training on ImageNet compared to SGD and gossip-based methods.
Distributed training algorithms of deep neural networks show impressive convergence speedup properties on very large problems. However, they inherently suffer from communication related slowdowns and communication topology becomes a crucial design choice. Common approaches supported by most machine learning frameworks are: 1) Synchronous decentralized algorithms relying on a peer-to-peer All Reduce topology that is sensitive to stragglers and communication delays. 2) Asynchronous centralised algorithms with a server based topology that is prone to communication bottleneck. Researchers also suggested asynchronous decentralized algorithms designed to avoid the bottleneck and speedup training, however, those commonly use inexact sparse averaging that may lead to a degradation in accuracy. In this paper, we propose Local Asynchronous SGD (LASGD), an asynchronous decentralized algorithm that relies on All Reduce for model synchronization. We empirically validate LASGD's performance on image classification tasks on the ImageNet dataset. Our experiments demonstrate that LASGD accelerates training compared to SGD and state of the art gossip based approaches.