Benign Overfitting in Two-layer Convolutional Neural Networks
This provides theoretical insights into overfitting in neural networks, addressing a gap for researchers in machine learning theory, though it is incremental by extending prior work from linear models to CNNs.
The paper tackles the problem of understanding when benign overfitting occurs in neural networks by studying a two-layer convolutional neural network, showing that a sharp phase transition between benign and harmful overfitting is driven by the signal-to-noise ratio, with arbitrarily small test loss achievable under certain conditions.
Modern neural networks often have great expressive power and can be trained to overfit the training data, while still achieving a good test performance. This phenomenon is referred to as "benign overfitting". Recently, there emerges a line of works studying "benign overfitting" from the theoretical perspective. However, they are limited to linear models or kernel/random feature models, and there is still a lack of theoretical understanding about when and how benign overfitting occurs in neural networks. In this paper, we study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN). We show that when the signal-to-noise ratio satisfies a certain condition, a two-layer CNN trained by gradient descent can achieve arbitrarily small training and test loss. On the other hand, when this condition does not hold, overfitting becomes harmful and the obtained CNN can only achieve a constant level test loss. These together demonstrate a sharp phase transition between benign overfitting and harmful overfitting, driven by the signal-to-noise ratio. To the best of our knowledge, this is the first work that precisely characterizes the conditions under which benign overfitting can occur in training convolutional neural networks.