Sequential convergence of AdaGrad algorithm for smooth convex optimization
This work provides a theoretical guarantee of convergence for the AdaGrad algorithm, which is important for researchers and practitioners using this optimization method.
This paper demonstrates that the iterates of both scalar and coordinate-wise AdaGrad algorithms converge when applied to convex objective functions with Lipschitz gradients. The convergence is established by identifying a variable metric quasi-Fejér monotonicity property in the AdaGrad sequences.
We prove that the iterates produced by, either the scalar step size variant, or the coordinatewise variant of AdaGrad algorithm, are convergent sequences when applied to convex objective functions with Lipschitz gradient. The key insight is to remark that such AdaGrad sequences satisfy a variable metric quasi-Fejér monotonicity property, which allows to prove convergence.