On regularization of gradient descent, layer imbalance and flat minima
This work provides theoretical insights into regularization mechanisms in deep learning, but it is incremental as it builds on existing concepts of flat minima and training dynamics.
The paper analyzes training dynamics in deep linear networks using a new metric called layer imbalance to define solution flatness, showing that different regularization methods behave similarly and training involves distinct optimization and regularization phases, with SGD acting like noise regularization.
We analyze the training dynamics for deep linear networks using a new metric - layer imbalance - which defines the flatness of a solution. We demonstrate that different regularization methods, such as weight decay or noise data augmentation, behave in a similar way. Training has two distinct phases: 1) optimization and 2) regularization. First, during the optimization phase, the loss function monotonically decreases, and the trajectory goes toward a minima manifold. Then, during the regularization phase, the layer imbalance decreases, and the trajectory goes along the minima manifold toward a flat area. Finally, we extend the analysis for stochastic gradient descent and show that SGD works similarly to noise regularization.