On the Acceleration of Deep Learning Model Parallelism with Staleness
This work addresses a critical performance bottleneck in distributed deep learning for computer vision, offering an incremental improvement over existing methods by handling multiple locking problems and stragglers.
The paper tackles the inefficiency of training large deep convolutional neural networks distributed across multiple devices, caused by locking problems and the straggler issue, by proposing the Diversely Stale Parameters (DSP) algorithm, which achieves significant training speedup with stronger robustness in experiments.
Training the deep convolutional neural network for computer vision problems is slow and inefficient, especially when it is large and distributed across multiple devices. The inefficiency is caused by the backpropagation algorithm's forward locking, backward locking, and update locking problems. Existing solutions for acceleration either can only handle one locking problem or lead to severe accuracy loss or memory inefficiency. Moreover, none of them consider the straggler problem among devices. In this paper, we propose Layer-wise Staleness and a novel efficient training algorithm, Diversely Stale Parameters (DSP), to address these challenges. We also analyze the convergence of DSP with two popular gradient-based methods and prove that both of them are guaranteed to converge to critical points for non-convex problems. Finally, extensive experimental results on training deep learning models demonstrate that our proposed DSP algorithm can achieve significant training speedup with stronger robustness than compared methods.