Noisy Interpolation Learning with Shallow Univariate ReLU Networks
This provides foundational insights into overfitting behavior in neural networks, addressing a key theoretical gap in machine learning.
The paper tackles the problem of understanding how overparameterized neural networks generalize with noisy training data, specifically analyzing two-layer ReLU networks and showing that overfitting is tempered for L1 loss but catastrophic for L2 loss or in expectation.
Understanding how overparameterized neural networks generalize despite perfect interpolation of noisy training data is a fundamental question. Mallinar et. al. 2022 noted that neural networks seem to often exhibit ``tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error, but neither does it approach infinity, yielding non-trivial generalization. However, this has not been studied rigorously. We provide the first rigorous analysis of the overfitting behavior of regression with minimum norm ($\ell_2$ of weights), focusing on univariate two-layer ReLU networks. We show overfitting is tempered (with high probability) when measured with respect to the $L_1$ loss, but also show that the situation is more complex than suggested by Mallinar et. al., and overfitting is catastrophic with respect to the $L_2$ loss, or when taking an expectation over the training set.