Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation
For NLP researchers studying annotation variation, this work provides the first large-scale analysis of interactions between linguistic features and annotator characteristics, but findings are dataset-specific and incremental.
This paper analyzes how linguistic properties of items and annotator characteristics interact to cause annotation variation in harmful language detection, finding that interactions reveal intersectional effects ignored in prior work and that lexical cues and annotator attitudes play a strong role, though patterns vary across datasets.
Human label variation has been established as a central phenomenon in NLP: the perspectives different annotators have on the same item need to be embraced. Data collection practices thus shifted towards increasing the annotator numbers and releasing disaggregated datasets, harmful language being most resourced due to its high subjectivity. While this resulted in rich information about \textit{who} annotated (sociodemographics, attitudes, etc.), the \textit{what} (e.g., linguistic properties of items), and their interplay has received little attention. We present the first large-scale analysis of four reference datasets for harmful language detection, bringing together annotator characteristics, linguistic properties of the items, and their interactions in a statistically informed picture. We find that interactions are crucial, revealing intersectional effects ignored in previous work, and that a strong role is played by lexical cues and annotator attitudes. Effect patterns, however, vary considerably across datasets. This urges caution about generalization and transferability.