Loss Functions Robust to the Presence of Label Errors
This work addresses the challenge of label noise in training data for machine learning practitioners, but it is incremental as it builds upon existing loss function adjustments like Focal Loss.
The paper tackles the problem of training models robust to label errors by proposing two novel loss functions that de-weight or ignore difficult samples likely to have label errors, resulting in improved F1 scores for error detection compared to baselines like categorical Cross Entropy and Focal Loss on artificially corrupted data.
Methods for detecting label errors in training data require models that are robust to label errors (i.e., not fit to erroneously labelled data points). However, acquiring such models often involves training on corrupted data, which presents a challenge. Adjustments to the loss function present an opportunity for improvement. Motivated by Focal Loss (which emphasizes difficult-to-classify samples), two novel, yet simple, loss functions are proposed that de-weight or ignore these difficult samples (i.e., those likely to have label errors). Results on artificially corrupted data show promise, such that F1 scores for detecting errors are improved from the baselines of conventional categorical Cross Entropy and Focal Loss.