Ghost Loss to Question the Reliability of Training Data
This work addresses the issue of annotation errors in training data for deep learning practitioners, which can lead to reduced model performance and unnecessary complexity, though it is incremental as it builds on existing loss function concepts.
The authors tackled the problem of unreliable training data annotations in supervised image classification by introducing ghost loss, which allows networks to ignore potentially incorrect labels without penalty, and they demonstrated its ability to detect confusing and erroneously labeled images in datasets like MNIST and CIFAR10.
Supervised image classification problems rely on training data assumed to have been correctly annotated; this assumption underpins most works in the field of deep learning. In consequence, during its training, a network is forced to match the label provided by the annotator and is not given the flexibility to choose an alternative to inconsistencies that it might be able to detect. Therefore, erroneously labeled training images may end up ``correctly'' classified in classes which they do not actually belong to. This may reduce the performances of the network and thus incite to build more complex networks without even checking the quality of the training data. In this work, we question the reliability of the annotated datasets. For that purpose, we introduce the notion of ghost loss, which can be seen as a regular loss that is zeroed out for some predicted values in a deterministic way and that allows the network to choose an alternative to the given label without being penalized. After a proof of concept experiment, we use the ghost loss principle to detect confusing images and erroneously labeled images in well-known training datasets (MNIST, Fashion-MNIST, SVHN, CIFAR10) and we provide a new tool, called sanity matrix, for summarizing these confusions.