On a Conjecture Regarding the Adam Optimizer
This work resolves a theoretical gap in understanding Adam's effectiveness, which is incremental but important for researchers and practitioners in deep learning.
The paper addresses a missing piece in the mathematical justification for the Adam optimizer by disproving Bock's conjecture and proving a modified version, which provides a corrected theoretical foundation for analyzing Adam's performance.
Why does the Adam optimizer work so well in deep-learning applications? Adam's originators, Kingma and Ba, presented a mathematical argument that was meant to help explain its success, but Bock and colleagues have since reported that a key piece is missing from that argument $-$ an unproven lemma which we will call Bock's conjecture. Here we show that this conjecture is false, but we prove a modified version of it $-$ a generalization of a result of Reddi and colleagues $-$ which can take its place in analyses of Adam.