FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information
This provides theoretical foundations and practical improvements for the widely-used Adam optimizer, though it's incremental rather than paradigm-shifting.
The paper establishes Adam's connection to natural gradient descent using diagonal empirical Fisher information, identifies flaws in the original algorithm, and proposes corrections including enhanced momentum and adaptive epsilon. Their modified FAdam algorithm achieves state-of-the-art results in ASR and superior performance across LLM, ASR, and VQ-VAE domains.
This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We provide an accessible and detailed analysis of the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, adaptive epsilon, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.