GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
This addresses the need for faster training of large-scale neural networks in computer vision, but it appears incremental as it combines existing techniques.
The paper tackles the problem of slow neural network training by combining model parallelism (GPU acceleration) and data parallelism (asynchronous stochastic gradient descent), reporting that GPU A-SGD can speed up training for large convolutional neural networks in computer vision.
The ability to train large-scale neural networks has resulted in state-of-the-art performance in many areas of computer vision. These results have largely come from computational break throughs of two forms: model parallelism, e.g. GPU accelerated training, which has seen quick adoption in computer vision circles, and data parallelism, e.g. A-SGD, whose large scale has been used mostly in industry. We report early experiments with a system that makes use of both model parallelism and data parallelism, we call GPU A-SGD. We show using GPU A-SGD it is possible to speed up training of large convolutional neural networks useful for computer vision. We believe GPU A-SGD will make it possible to train larger networks on larger training sets in a reasonable amount of time.