Pre-train to Gain: Robust Learning Without Clean Labels
This addresses the challenge of noisy labels in machine learning, offering a practical solution for scenarios where clean data is unavailable, though it is incremental by building on existing self-supervised methods.
The paper tackles the problem of training deep networks with noisy labels by using self-supervised pre-training to improve robustness without needing clean data. It shows that this approach consistently boosts classification accuracy and label-error detection across noise rates, outperforming ImageNet pre-trained models under high noise conditions.
Training deep networks with noisy labels leads to poor generalization and degraded accuracy due to overfitting to label noise. Existing approaches for learning with noisy labels often rely on the availability of a clean subset of data. By pre-training a feature extractor backbone without labels using self-supervised learning (SSL), followed by standard supervised training on the noisy dataset, we can train a more noise robust model without requiring a subset with clean labels. We evaluate the use of SimCLR and Barlow~Twins as SSL methods on CIFAR-10 and CIFAR-100 under synthetic and real world noise. Across all noise rates, self-supervised pre-training consistently improves classification accuracy and enhances downstream label-error detection (F1 and Balanced Accuracy). The performance gap widens as the noise rate increases, demonstrating improved robustness. Notably, our approach achieves comparable results to ImageNet pre-trained models at low noise levels, while substantially outperforming them under high noise conditions.