Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation
This addresses the need for standardized agreement measurement in NLP annotation, which is crucial for reliable data but often handled inconsistently across tasks.
This paper tackles the problem of selecting appropriate inter-annotator agreement metrics for diverse NLP annotation tasks, organizing measures by task type and discussing factors like label imbalance. It provides guidelines for clear reporting and interpretation to improve consistency and reproducibility in human annotation.
Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and continuous rating, measuring agreement between annotators has become increasingly more complex. This paper outlines how inter-annotator agreement (IAA) has been conceptualised and applied across NLP and related disciplines, describing the assumptions and limitations of common approaches. We organise agreement measures by task type and discuss how factors such as label imbalance and missing data influence reliability estimates. In addition, we highlight best practices for clear and transparent reporting, including the use of confidence intervals and the analysis of disagreement patterns. The paper aims to serve as a guide for selecting and interpreting agreement measures, promoting more consistent and reproducible human annotation and evaluation in NLP.