Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
This work addresses the lack of controlled datasets for real-world label noise, providing a benchmark for researchers to better understand and improve deep learning robustness in noisy environments.
The paper tackles the problem of studying deep learning with controlled real-world label noise by establishing the first benchmark for web label noise, enabling controlled experiments for the first time. It introduces a simple method that achieves state-of-the-art results on their dataset and public benchmarks like CIFAR and WebVision.
Performing controlled experiments on noisy data is essential in understanding deep learning across noise levels. Due to the lack of suitable datasets, previous research has only examined deep learning on controlled synthetic label noise, and real-world label noise has never been studied in a controlled setting. This paper makes three contributions. First, we establish the first benchmark of controlled real-world label noise from the web. This new benchmark enables us to study the web label noise in a controlled setting for the first time. The second contribution is a simple but effective method to overcome both synthetic and real noisy labels. We show that our method achieves the best result on our dataset as well as on two public benchmarks (CIFAR and WebVision). Third, we conduct the largest study by far into understanding deep neural networks trained on noisy labels across different noise levels, noise types, network architectures, and training settings. The data and code are released at the following link: http://www.lujiang.info/cnlw.html