Two Wrongs Don't Make a Right: Combating Confirmation Bias in Learning with Label Noise
This work addresses a critical issue in robust deep learning for noisy labels, offering a solution that mitigates confirmation bias and improves performance, though it is incremental as it builds on existing two-stage pipelines.
The paper tackles the problem of confirmation bias in two-stage methods for learning with noisy labels, where pseudo-labels fail to correct many noisy labels, leading to error accumulation. It proposes Robust Label Refurbishment, a hybrid method that integrates pseudo-labeling and confidence estimation to refurbish noisy labels, achieving state-of-the-art performance across datasets like CIFAR, Mini-WebVision, and ANIMAL-10N under various noise types.
Noisy labels damage the performance of deep networks. For robust learning, a prominent two-stage pipeline alternates between eliminating possible incorrect labels and semi-supervised training. However, discarding part of noisy labels could result in a loss of information, especially when the corruption has a dependency on data, e.g., class-dependent or instance-dependent. Moreover, from the training dynamics of a representative two-stage method DivideMix, we identify the domination of confirmation bias: pseudo-labels fail to correct a considerable amount of noisy labels, and consequently, the errors accumulate. To sufficiently exploit information from noisy labels and mitigate wrong corrections, we propose Robust Label Refurbishment (Robust LR) a new hybrid method that integrates pseudo-labeling and confidence estimation techniques to refurbish noisy labels. We show that our method successfully alleviates the damage of both label noise and confirmation bias. As a result, it achieves state-of-the-art performance across datasets and noise types, namely CIFAR under different levels of synthetic noise and Mini-WebVision and ANIMAL-10N with real-world noise.