Time-Delay Momentum: A Regularization Perspective on the Convergence and Generalization of Stochastic Momentum for Deep Learning
This work addresses convergence and generalization issues for deep learning practitioners, offering a novel regularization perspective with proven bounds, though it is incremental as it builds on existing momentum methods.
The paper tackles the problem of analyzing convergence and generalization for stochastic momentum in deep learning by interpreting it as a regularization method, proposing time-delay momentum, and proves convergence rates of O(1/√K) and generalization bounds of O(1/√nδ) with empirical superiority over state-of-the-art solvers.
In this paper we study the problem of convergence and generalization error bound of stochastic momentum for deep learning from the perspective of regularization. To do so, we first interpret momentum as solving an $\ell_2$-regularized minimization problem to learn the offsets between arbitrary two successive model parameters. We call this {\em time-delay momentum} because the model parameter is updated after a few iterations towards finding the minimizer. We then propose our learning algorithm, \ie stochastic gradient descent (SGD) with time-delay momentum. We show that our algorithm can be interpreted as solving a sequence of strongly convex optimization problems using SGD. We prove that under mild conditions our algorithm can converge to a stationary point with rate of $O(\frac{1}{\sqrt{K}})$ and generalization error bound of $O(\frac{1}{\sqrt{nδ}})$ with probability at least $1-δ$, where $K,n$ are the numbers of model updates and training samples, respectively. We demonstrate the empirical superiority of our algorithm in deep learning in comparison with the state-of-the-art deep learning solvers.