Implicit Gradient Regularization
This work provides theoretical insight into the interaction of learning rate, model size, and regularization in overparameterized models, which is incremental but clarifies a fundamental aspect of deep learning optimization.
The paper tackles the problem of understanding why gradient descent optimizes deep neural networks effectively without explicit regularization, finding that discrete gradient descent steps implicitly penalize large loss gradients, which biases optimization toward flat minima and improves test error robustness.
Gradient descent can be surprisingly good at optimizing deep neural networks without overfitting and without explicit regularization. We find that the discrete steps of gradient descent implicitly regularize models by penalizing gradient descent trajectories that have large loss gradients. We call this Implicit Gradient Regularization (IGR) and we use backward error analysis to calculate the size of this regularization. We confirm empirically that implicit gradient regularization biases gradient descent toward flat minima, where test errors are small and solutions are robust to noisy parameter perturbations. Furthermore, we demonstrate that the implicit gradient regularization term can be used as an explicit regularizer, allowing us to control this gradient regularization directly. More broadly, our work indicates that backward error analysis is a useful theoretical approach to the perennial question of how learning rate, model size, and parameter regularization interact to determine the properties of overparameterized models optimized with gradient descent.