Curriculum Loss: Robust Learning and Generalization against Label Corruption
This addresses robustness and generalization against label corruption in deep learning, which is an incremental improvement in loss function design.
The paper tackles the problem of deep neural networks memorizing wrong labels by proposing curriculum loss (CL), a simple and efficient loss that is a tighter upper bound of the 0-1 loss and adaptively selects samples for training, with experimental results on benchmark datasets validating its robustness.
Deep neural networks (DNNs) have great expressive power, which can even memorize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper studies the 0-1 loss, which has a monotonic relationship with an empirical adversary (reweighted) risk~\citep{hu2016does}. Although the 0-1 loss has some robust properties, it is difficult to optimize. To efficiently optimize the 0-1 loss while keeping its robust properties, we propose a very simple and efficient loss, i.e. curriculum loss (CL). Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses. Moreover, CL can adaptively select samples for model training. As a result, our loss can be deemed as a novel perspective of curriculum sample selection strategy, which bridges a connection between curriculum learning and robust learning. Experimental results on benchmark datasets validate the robustness of the proposed loss.