Convergence rates and approximation results for SGD and its continuous-time counterpart
This work offers incremental theoretical improvements for researchers analyzing optimization algorithms, with potential applications in machine learning optimization theory.
This paper provides a theoretical analysis of Stochastic Gradient Descent (SGD) with non-increasing step sizes, showing it can be approximated by a stochastic differential equation and establishing improved non-asymptotic convergence bounds for convex and some non-convex functions.
This paper proposes a thorough theoretical analysis of Stochastic Gradient Descent (SGD) with non-increasing step sizes. First, we show that the recursion defining SGD can be provably approximated by solutions of a time inhomogeneous Stochastic Differential Equation (SDE) using an appropriate coupling. In the specific case of a batch noise we refine our results using recent advances in Stein's method. Then, motivated by recent analyses of deterministic and stochastic optimization methods by their continuous counterpart, we study the long-time behavior of the continuous processes at hand and establish non-asymptotic bounds. To that purpose, we develop new comparison techniques which are of independent interest. Adapting these techniques to the discrete setting, we show that the same results hold for the corresponding SGD sequences. In our analysis, we notably improve non-asymptotic bounds in the convex setting for SGD under weaker assumptions than the ones considered in previous works. Finally, we also establish finite-time convergence results under various conditions, including relaxations of the famous Łojasiewicz inequality, which can be applied to a class of non-convex functions.