Detecting Label Errors in Token Classification Data
This addresses the issue of mislabeled examples in token classification data, which is a common problem in real-world applications, but the approach is incremental as it builds on existing methods.
The paper tackled the problem of detecting label errors in token classification datasets by evaluating 11 straightforward methods based on predicted class probabilities from token classification models, and identified a simple and effective method that consistently detects sentences containing label errors in CoNLL-2003 entity recognition data.
Mislabeled examples are a common issue in real-world data, particularly for tasks like token classification where many labels must be chosen on a fine-grained basis. Here we consider the task of finding sentences that contain label errors in token classification datasets. We study 11 different straightforward methods that score tokens/sentences based on the predicted class probabilities output by a (any) token classification model (trained via any procedure). In precision-recall evaluations based on real-world label errors in entity recognition data from CoNLL-2003, we identify a simple and effective method that consistently detects those sentences containing label errors when applied with different token classification models.