Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach
This addresses the challenge of label noise in deep learning, which is critical for real-world applications where data may be mislabeled, but the approach is incremental as it builds on existing noise estimation techniques.
The paper tackles the problem of training deep neural networks robustly under class-dependent label noise by proposing loss correction procedures, demonstrating effectiveness across diverse datasets and architectures with extensive experiments.
We present a theoretically grounded approach to train deep neural networks, including recurrent networks, subject to class-dependent label noise. We propose two procedures for loss correction that are agnostic to both application domain and network architecture. They simply amount to at most a matrix inversion and multiplication, provided that we know the probability of each class being corrupted into another. We further show how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and thus providing an end-to-end framework. Extensive experiments on MNIST, IMDB, CIFAR-10, CIFAR-100 and a large scale dataset of clothing images employing a diversity of architectures --- stacking dense, convolutional, pooling, dropout, batch normalization, word embedding, LSTM and residual layers --- demonstrate the noise robustness of our proposals. Incidentally, we also prove that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise.