Deep linear networks for regression are implicitly regularized towards flat minima
This work addresses the theoretical understanding of implicit regularization in neural networks, which is incremental but provides specific insights into flat minima for researchers in optimization and deep learning.
The paper tackles the problem of understanding optimization dynamics in deep linear networks for regression by analyzing the sharpness (largest Hessian eigenvalue) of minimizers, showing that while sharpness can be arbitrarily large, there is a lower bound that grows linearly with depth, and gradient flow implicitly regularizes towards flat minima with sharpness bounded by a constant times this lower bound, independent of width or depth.
The largest eigenvalue of the Hessian, or sharpness, of neural networks is a key quantity to understand their optimization dynamics. In this paper, we study the sharpness of deep linear networks for univariate regression. Minimizers can have arbitrarily large sharpness, but not an arbitrarily small one. Indeed, we show a lower bound on the sharpness of minimizers, which grows linearly with depth. We then study the properties of the minimizer found by gradient flow, which is the limit of gradient descent with vanishing learning rate. We show an implicit regularization towards flat minima: the sharpness of the minimizer is no more than a constant times the lower bound. The constant depends on the condition number of the data covariance matrix, but not on width or depth. This result is proven both for a small-scale initialization and a residual initialization. Results of independent interest are shown in both cases. For small-scale initialization, we show that the learned weight matrices are approximately rank-one and that their singular vectors align. For residual initialization, convergence of the gradient flow for a Gaussian initialization of the residual network is proven. Numerical experiments illustrate our results and connect them to gradient descent with non-vanishing learning rate.