Distributed Training of Deep Neural Networks with Theoretical Analysis: Under SSP Setting
This work addresses the challenge of scaling deep learning training efficiently for researchers and practitioners, though it appears incremental as it builds on existing distributed methods with added theoretical analysis.
The authors tackled the problem of distributed training for deep neural networks by proposing a method with theoretical convergence guarantees, achieving close to 6 times faster training on ImageNet with 6 machines and showing scalability across datasets like TIMIT and ImageNet for tasks such as image classification and phoneme extraction.
We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, and guaranteed to converge to the same optima as the undistributed setting. The convergence and scalability of the distributed setting is shown empirically across different datasets (TIMIT and ImageNet) and machine learning tasks (image classification and phoneme extraction). The convergence analysis provides novel insights into this complex learning scheme, including: 1) layerwise convergence, and 2) convergence of the weights in probability.