Error estimates between SGD with momentum and underdamped Langevin diffusion
This work addresses a theoretical gap for researchers in optimization and machine learning, offering incremental insights into the convergence properties of these algorithms.
The paper tackles the problem of quantifying the relationship between stochastic gradient descent with momentum and underdamped Langevin diffusion by establishing error estimates in 1-Wasserstein and total variation distances, providing concrete mathematical bounds.
Stochastic gradient descent with momentum is a popular variant of stochastic gradient descent, which has recently been reported to have a close relationship with the underdamped Langevin diffusion. In this paper, we establish a quantitative error estimate between them in the 1-Wasserstein and total variation distances.