The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks
This work provides theoretical insights into a fundamental phenomenon in deep learning, but it is incremental as it builds on existing results about implicit bias.
The paper tackles the problem of understanding benign overfitting in two-layer linear neural networks by deriving excess risk bounds under sub-Gaussian and anti-concentration conditions, showing that initialization quality and data covariance properties are key factors in achieving low risk.
The recent success of neural network models has shone light on a rather surprising statistical phenomenon: statistical models that perfectly fit noisy data can generalize well to unseen test data. Understanding this phenomenon of $\textit{benign overfitting}$ has attracted intense theoretical and empirical study. In this paper, we consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk when the covariates satisfy sub-Gaussianity and anti-concentration properties, and the noise is independent and sub-Gaussian. By leveraging recent results that characterize the implicit bias of this estimator, our bounds emphasize the role of both the quality of the initialization as well as the properties of the data covariance matrix in achieving low excess risk.