From Tempered to Benign Overfitting in ReLU Neural Networks
This work addresses a theoretical gap in machine learning by clarifying how input dimension influences overfitting behavior in neural networks, providing insights for researchers studying generalization.
The paper tackles the problem of understanding overfitting in overparameterized neural networks, proving that for 2-layer ReLU networks, the type of overfitting transitions from tempered (non-optimal performance degrading with noise) in one-dimensional data to benign (near-optimal performance) in high dimensions, with empirical validation for intermediate dimensions.
Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign overfitting", where interpolating predictors achieve near-optimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting", where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of this claim for non-linear NNs has been lacking so far. In this work, we provide several results that aim at bridging these complementing views. We study a simple classification setting with 2-layer ReLU NNs, and prove that under various assumptions, the type of overfitting transitions from tempered in the extreme case of one-dimensional data, to benign in high dimensions. Thus, we show that the input dimension has a crucial role on the type of overfitting in this setting, which we also validate empirically for intermediate dimensions. Overall, our results shed light on the intricate connections between the dimension, sample size, architecture and training algorithm on the one hand, and the type of resulting overfitting on the other hand.