Dimension-free convergence rates for gradient Langevin dynamics in RKHS
This work addresses a key bottleneck for practitioners in machine learning by providing convergence guarantees that scale independently of dimension, which is crucial for high-dimensional or infinite-dimensional applications like kernel methods.
The authors tackled the problem of exponential dimension-dependence in convergence rates for gradient Langevin dynamics (GLD) and stochastic GLD (SGLD) in non-convex optimization, deriving non-asymptotic, dimension-free convergence rates for these methods in infinite-dimensional reproducing kernel Hilbert spaces.
Gradient Langevin dynamics (GLD) and stochastic GLD (SGLD) have attracted considerable attention lately, as a way to provide convergence guarantees in a non-convex setting. However, the known rates grow exponentially with the dimension of the space. In this work, we provide a convergence analysis of GLD and SGLD when the optimization space is an infinite dimensional Hilbert space. More precisely, we derive non-asymptotic, dimension-free convergence rates for GLD/SGLD when performing regularized non-convex optimization in a reproducing kernel Hilbert space. Amongst others, the convergence analysis relies on the properties of a stochastic differential equation, its discrete time Galerkin approximation and the geometric ergodicity of the associated Markov chains.