Central limit theorem for the averaged Adam optimizer
For practitioners and theorists using Adam, this offers a precise asymptotic characterization of its convergence, though it is an incremental theoretical analysis of an existing algorithm.
This paper provides a central limit theorem for the averaged Adam optimizer, quantifying its convergence speed as n^{-1/2}, matching classical stochastic approximation algorithms. The covariance is expressed in terms of Adam's properties at the attractor.
In this article, we analyse convergence of the averaged Adam optimizer to an attracting zero of the Adam vector field. We provide a central limit theorem that, in particular, quantifies exactly the speed of convergence. The order of convergence is $n^{-1/2}$ in the number of steps of the algorithm which coincides with the order observed for classical stochastic approximation algorithms. The covariance in the central limit theorem is given in terms of properties of the Adam algorithm in the state of the attractor.