On Higher-order Moments in Adam
This work offers an incremental improvement for deep learning practitioners by enhancing a widely used optimization algorithm.
The authors tackled the problem of improving the Adam optimizer by extending it to use higher-order moments of the stochastic gradient, resulting in HAdam, which achieved better performance than vanilla Adam in experiments.
In this paper, we investigate the popular deep learning optimization routine, Adam, from the perspective of statistical moments. While Adam is an adaptive lower-order moment based (of the stochastic gradient) method, we propose an extension namely, HAdam, which uses higher order moments of the stochastic gradient. Our analysis and experiments reveal that certain higher-order moments of the stochastic gradient are able to achieve better performance compared to the vanilla Adam algorithm. We also provide some analysis of HAdam related to odd and even moments to explain some intriguing and seemingly non-intuitive empirical results.