Divergence Results and Convergence of a Variance Reduced Version of ADAM
This work tackles convergence issues in widely used optimization algorithms like ADAM, offering a potential fix for training deep neural networks, though it is incremental as it builds on existing methods.
The paper addresses the divergence of ADAM in stochastic optimization by providing stronger divergence examples and proposing a variance-reduced version that converges under reduced gradient variance assumptions, with numerical experiments showing comparable performance to ADAM.
Stochastic optimization algorithms using exponential moving averages of the past gradients, such as ADAM, RMSProp and AdaGrad, have been having great successes in many applications, especially in training deep neural networks. ADAM in particular stands out as efficient and robust. Despite of its outstanding performance, ADAM has been proved to be divergent for some specific problems. We revisit the divergent question and provide divergent examples under stronger conditions such as in expectation or high probability. Under a variance reduction assumption, we show that an ADAM-type algorithm converges, which means that it is the variance of gradients that causes the divergence of original ADAM. To this end, we propose a variance reduced version of ADAM and provide a convergent analysis of the algorithm. Numerical experiments show that the proposed algorithm has as good performance as ADAM. Our work suggests a new direction for fixing the convergence issues.