Gossip training for deep learning
This work addresses the need for faster training in deep learning, though it appears incremental as it builds on existing distributed SGD methods.
The paper tackles the problem of accelerating convolutional network training by proposing GoSGD, a fully asynchronous and decentralized distributed method based on gossip algorithms, which shows encouraging results compared to EASGD on CIFAR-10.
We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information between different threads inspired by gossip algorithms and showing good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized. We compared our method to the recent EASGD in \cite{elastic} on CIFAR-10 show encouraging results.