How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?
This addresses generalization issues in deep learning for noisy data, offering a theoretical solution with potential broad impact, though it is incremental based on prior observations of label noise regularization.
The paper tackles the problem of neural networks overfitting to noise in low signal-to-noise ratio (SNR) data, showing that adding label noise to gradient descent suppresses noise memorization and improves generalization, with proven lower bounds on test error for standard gradient descent.
The capacity of deep learning models is often large enough to both learn the underlying statistical signal and overfit to noise in the training set. This noise memorization can be harmful especially for data with a low signal-to-noise ratio (SNR), leading to poor generalization. Inspired by prior observations that label noise provides implicit regularization that improves generalization, in this work, we investigate whether introducing label noise to the gradient updates can enhance the test performance of neural network (NN) in the low SNR regime. Specifically, we consider training a two-layer NN with a simple label noise gradient descent (GD) algorithm, in an idealized signal-noise data setting. We prove that adding label noise during training suppresses noise memorization, preventing it from dominating the learning process; consequently, label noise GD enjoys rapid signal growth while the overfitting remains controlled, thereby achieving good generalization despite the low SNR. In contrast, we also show that NN trained with standard GD tends to overfit to noise in the same low SNR setting and establish a non-vanishing lower bound on its test error, thus demonstrating the benefit of introducing label noise in gradient-based training.