A Random Matrix Theory Approach to Damping in Deep Learning
This addresses a fundamental optimization problem in deep learning for researchers and practitioners, offering a method to improve generalization in adaptive methods, though it is incremental as it builds on existing second-order optimizer concepts.
The paper tackles the generalization gap between adaptive and non-adaptive gradient methods in deep learning by attributing it to increased noise in flat loss surface directions, and shows that increasing a shrinkage coefficient closes this gap entirely in logistic regression and deep neural networks, with a novel random matrix theory-based damping learner enabling fast convergence and competitive generalization.
We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the flattest directions of the true loss surface. We demonstrate that typical schedules used for adaptive methods (with low numerical stability or damping constants) serve to bias relative movement towards flat directions relative to sharp directions, effectively amplifying the noise-to-signal ratio and harming generalisation. We further demonstrate that the numerical damping constant used in these methods can be decomposed into a learning rate reduction and linear shrinkage of the estimated curvature matrix. We then demonstrate significant generalisation improvements by increasing the shrinkage coefficient, closing the generalisation gap entirely in both logistic regression and several deep neural network experiments. Extending this line further, we develop a novel random matrix theory based damping learner for second order optimiser inspired by linear shrinkage estimation. We experimentally demonstrate our learner to be very insensitive to the initialised value and to allow for extremely fast convergence in conjunction with continued stable training and competitive generalisation.