Preconditioning for Accelerated Gradient Descent Optimization and Regularization
This work addresses a theoretical gap for machine learning practitioners by explaining how common optimization and regularization methods interact, though it is incremental as it builds on existing theories without introducing new algorithms.
The paper tackles the challenge of understanding and combining regularization with preconditioning in accelerated training algorithms, showing that methods like AdamW and normalization improve Hessian conditioning and providing a unified mathematical framework for these techniques.
Accelerated training algorithms, such as adaptive learning rates (or preconditioning) and various normalization methods, are widely used but not fully understood. When regularization is introduced, standard optimizers like adaptive learning rates may not perform effectively. This raises the need for alternative regularization approaches such as AdamW and the question of how to properly combine regularization with preconditioning. In this paper, we address these challenges using the theory of preconditioning as follows: (1) We explain how AdaGrad, RMSProp, and Adam accelerates training through improving Hessian conditioning; (2) We explore the interaction between $L_2$-regularization and preconditioning, demonstrating that AdamW amounts to selecting the underlying intrinsic parameters for regularization, and we derive a generalization for the $L_1$-regularization; and (3) We demonstrate how various normalization methods such as input data normalization, batch normalization, and layer normalization accelerate training by improving Hessian conditioning. Our analysis offers a unified mathematical framework for understanding various acceleration techniques or deriving appropriate regularization schemes.