Benign Overfitting for Two-layer ReLU Convolutional Neural Networks
This work addresses a foundational open problem in deep learning theory by providing theoretical insights into when benign overfitting occurs in non-smooth ReLU networks, which is incremental as it builds on prior studies limited to smooth activations or neural tangent kernels.
The paper tackles the problem of understanding benign overfitting in ReLU neural networks, establishing that two-layer ReLU convolutional networks trained by gradient descent can achieve near-zero training loss and Bayes optimal test risk under mild conditions, with experiments on synthetic data supporting the theory.
Modern deep learning models with great expressive power can be trained to overfit the training data but still generalize well. This phenomenon is referred to as \textit{benign overfitting}. Recently, a few studies have attempted to theoretically understand benign overfitting in neural networks. However, these works are either limited to neural networks with smooth activation functions or to the neural tangent kernel regime. How and when benign overfitting can occur in ReLU neural networks remains an open problem. In this work, we seek to answer this question by establishing algorithm-dependent risk bounds for learning two-layer ReLU convolutional neural networks with label-flipping noise. We show that, under mild conditions, the neural network trained by gradient descent can achieve near-zero training loss and Bayes optimal test risk. Our result also reveals a sharp transition between benign and harmful overfitting under different conditions on data distribution in terms of test risk. Experiments on synthetic data back up our theory.