CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset
This provides a more reliable benchmark for NER researchers to objectively compare methods and analyze errors, though it's incremental as it improves an existing dataset rather than introducing new methods.
The authors tackled annotation errors in the CoNLL-03 NER dataset by creating CleanCoNLL, a relabeled version that corrects 7.0% of labels, resulting in state-of-the-art models achieving 97.1% F1-score and reducing false error counts from 47% to 6%.
The CoNLL-03 corpus is arguably the most well-known and utilized benchmark dataset for named entity recognition (NER). However, prior works found significant numbers of annotation errors, incompleteness, and inconsistencies in the data. This poses challenges to objectively comparing NER approaches and analyzing their errors, as current state-of-the-art models achieve F1-scores that are comparable to or even exceed the estimated noise level in CoNLL-03. To address this issue, we present a comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our effort adds a layer of entity linking annotation both for better explainability of NER labels and as additional safeguard of annotation quality. Our experimental evaluation finds not only that state-of-the-art approaches reach significantly higher F1-scores (97.1%) on our data, but crucially that the share of correct predictions falsely counted as errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-art models, and that the theoretical upper bound even on high resource, coarse-grained NER is not yet reached. To facilitate such analysis, we make CleanCoNLL publicly available to the research community.