Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations
This addresses annotation inconsistencies in NER for researchers and practitioners, but it is incremental as it builds on existing studies of label variation.
The paper investigated sources of human label variation in named entity annotations across English, Danish, and Bavarian, finding that text ambiguity and guideline changes are dominant factors in high-quality datasets.
Named Entity Recognition (NER) is a key information extraction task with a long-standing tradition. While recent studies address and aim to correct annotation errors via re-labeling efforts, little is known about the sources of human label variation, such as text ambiguity, annotation error, or guideline divergence. This is especially the case for high-quality datasets and beyond English CoNLL03. This paper studies disagreements in expert-annotated named entity datasets for three languages: English, Danish, and Bavarian. We show that text ambiguity and artificial guideline changes are dominant factors for diverse annotations among high-quality revisions. We survey student annotations on a subset of difficult entities and substantiate the feasibility and necessity of manifold annotations for understanding named entity ambiguities from a distributional perspective.