Consistency Regularization Can Improve Robustness to Label Noise
This addresses the challenge of label noise in training data for machine learning practitioners, though it is incremental as it applies an existing regularization technique to a new robustness problem.
This paper tackles the problem of training neural networks with noisy labels by showing that consistency regularization improves robustness, achieving state-of-the-art results on synthetic datasets like CIFAR-10/100 and real-world WebVision data.
Consistency regularization is a commonly-used technique for semi-supervised and self-supervised learning. It is an auxiliary objective function that encourages the prediction of the network to be similar in the vicinity of the observed training samples. Hendrycks et al. (2020) have recently shown such regularization naturally brings test-time robustness to corrupted data and helps with calibration. This paper empirically studies the relevance of consistency regularization for training-time robustness to noisy labels. First, we make two interesting and useful observations regarding the consistency of networks trained with the standard cross entropy loss on noisy datasets which are: (i) networks trained on noisy data have lower consistency than those trained on clean data, and(ii) the consistency reduces more significantly around noisy-labelled training data points than correctly-labelled ones. Then, we show that a simple loss function that encourages consistency improves the robustness of the models to label noise on both synthetic (CIFAR-10, CIFAR-100) and real-world (WebVision) noise as well as different noise rates and types and achieves state-of-the-art results.