How does Early Stopping Help Generalization against Label Noise?
This addresses the issue of noisy labels in real-world data for machine learning practitioners, offering a practical solution that is not incremental but introduces a novel training approach.
The paper tackles the problem of poor generalization due to noisy labels in training data by proposing a two-phase method called Prestopping, which uses early stopping to avoid overfitting and then resumes training with a maximal safe set of likely true-labeled samples, achieving test error improvements of 0.4-8.2 percentage points over state-of-the-art methods on four image datasets.
Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we resume training the early stopped network using a "maximal safe set," which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point. Putting them all together, our novel two-phase training method, called Prestopping, realizes noise-free training under any type of label noise for practical use. Extensive experiments using four image benchmark data sets verify that our method significantly outperforms four state-of-the-art methods in test error by 0.4-8.2 percent points under existence of real-world noise.