Understanding the Generalization Benefits of Late Learning Rate Decay
This addresses a fundamental problem in deep learning optimization for researchers and practitioners, though it is incremental as it builds on existing observations about learning rates.
The paper investigates why neural networks trained with large learning rates for extended periods often generalize better, finding that this approach navigates training loss minima to approach testing loss minima, and demonstrates through a nonlinear model that it steers toward minimum norm solutions with near-optimal generalization.
Why do neural networks trained with large learning rates for a longer time often lead to better generalization? In this paper, we delve into this question by examining the relation between training and testing loss in neural networks. Through visualization of these losses, we note that the training trajectory with a large learning rate navigates through the minima manifold of the training loss, finally nearing the neighborhood of the testing loss minimum. Motivated by these findings, we introduce a nonlinear model whose loss landscapes mirror those observed for real neural networks. Upon investigating the training process using SGD on our model, we demonstrate that an extended phase with a large learning rate steers our model towards the minimum norm solution of the training loss, which may achieve near-optimal generalization, thereby affirming the empirically observed benefits of late learning rate decay.