Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
This provides theoretical insights into how optimization algorithms bias training in over-parameterized models, which is incremental as it builds on prior work on implicit regularization.
The paper tackles the problem of implicit regularization in over-parameterized models by analyzing discrete gradient dynamics in two-layer linear networks with least-squares loss, showing that with vanishing initialization and small step size, the dynamics sequentially learns reduced-rank regression solutions with increasing rank.
When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization algorithm and its respective hyper-parameters introduces biases that will lead to convergence to specific minimizers of the objective. Consequently, this choice can be considered as an implicit regularization for the training of over-parametrized models. In this work, we push this idea further by studying the discrete gradient dynamics of the training of a two-layer linear network with the least-squares loss. Using a time rescaling, we show that, with a vanishing initialization and a small enough step size, this dynamics sequentially learns the solutions of a reduced-rank regression with a gradually increasing rank.