Convergence of adaptive algorithms for weakly convex constrained optimization
This work addresses convergence guarantees for adaptive algorithms in constrained optimization, which is incremental as it extends existing results to a more general setting.
The paper tackles the problem of analyzing the convergence rate of the adaptive algorithm AMSGrad for constrained stochastic optimization with weakly convex objectives, proving a convergence rate of $\mathcal{ ilde O}(t^{-1/4})$ for the gradient norm of the Moreau envelope, which matches known rates for unconstrained cases.
We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the $\mathcal{\tilde O}(t^{-1/4})$ rate of convergence for the norm of the gradient of Moreau envelope, which is the standard stationarity measure for this class of problems. It matches the known rates that adaptive algorithms enjoy for the specific case of unconstrained smooth stochastic optimization. Our analysis works with mini-batch size of $1$, constant first and second order moment parameters, and possibly unbounded optimization domains. Finally, we illustrate the applications and extensions of our results to specific problems and algorithms.