Adam-like Algorithm with Smooth Clipping Attains Global Minima: Analysis Based on Ergodicity of Functional SDEs
This provides theoretical guarantees for a widely used optimization method in machine learning, addressing convergence in non-convex settings, though it is incremental as it builds on existing Adam-type algorithms.
The paper tackles the problem of proving global convergence for an Adam-type algorithm with smooth clipping on regularized non-convex loss functions, showing it attains the global minimizer with errors scaling as n^{-1/2}, η^{1/4}, β^{-1} log(β+1), and e^{-c t}.
In this paper, we prove that an Adam-type algorithm with smooth clipping approaches the global minimizer of the regularized non-convex loss function. Adding smooth clipping and taking the state space as the set of all trajectories, we can apply the ergodic theory of Markov semigroups for this algorithm and investigate its asymptotic behavior. The ergodic theory we establish in this paper reduces the problem of evaluating the convergence, generalization error and discretization error of this algorithm to the problem of evaluating the difference between two functional stochastic differential equations (SDEs) with different drift coefficients. As a result of our analysis, we have shown that this algorithm minimizes the the regularized non-convex loss function with errors of the form $n^{-1/2}$, $η^{1/4}$, $β^{-1} \log (β+ 1)$ and $e^{- c t}$. Here, $c$ is a constant and $n$, $η$, $β$ and $t$ denote the size of the training dataset, learning rate, inverse temperature and time, respectively.