Towards an Understanding of Benign Overfitting in Neural Networks
This addresses the theoretical puzzle of benign overfitting for researchers in machine learning, providing foundational insights into generalization despite overfitting.
The paper investigates how neural networks can overfit to noisy training data yet still generalize well, focusing on two-layer ReLU networks in high dimensions. It shows that under mild conditions, the excess risk decays and can achieve near minimax-optimal learning rates, with risk increasing when parameters exceed O(n^2).
Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss; yet surprisingly, they possess near-optimal prediction performance, contradicting classical learning theory. We examine how these benign overfitting phenomena occur in a two-layer neural network setting where sample covariates are corrupted with noise. We address the high dimensional regime, where the data dimension $d$ grows with the number $n$ of data points. Our analysis combines an upper bound on the bias with matching upper and lower bounds on the variance of the interpolator (an estimator that interpolates the data). These results indicate that the excess learning risk of the interpolator decays under mild conditions. We further show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate, which to our knowledge is the first generalization result for such networks. Finally, our theory predicts that the excess learning risk starts to increase once the number of parameters $s$ grows beyond $O(n^2)$, matching recent empirical findings.