A Unified Analysis of Stochastic Momentum Methods for Deep Learning
This provides theoretical insights for deep learning practitioners on when momentum methods are beneficial, though it is incremental as it builds on existing methods.
The paper tackled the lack of theoretical analysis for stochastic momentum methods in deep learning by unifying SG, SHB, and SNAG, finding that SHB and SNAG do not improve convergence over SG but can enhance generalization through improved stability, with empirical validation.
Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochastic gradient (SG) method, and the stochastic momentum methods including two famous variants, i.e., the stochastic heavy-ball (SHB) method and the stochastic variant of Nesterov's accelerated gradient (SNAG) method. We propose a framework that unifies the three variants. We then derive the convergence rates of the norm of gradient for the non-convex optimization problem, and analyze the generalization performance through the uniform stability approach. Particularly, the convergence analysis of the training objective exhibits that SHB and SNAG have no advantage over SG. However, the stability analysis shows that the momentum term can improve the stability of the learned model and hence improve the generalization performance. These theoretical insights verify the common wisdom and are also corroborated by our empirical analysis on deep learning.