Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Language
This addresses the problem of bias and variability in toxic language annotation for NLP researchers, though it is incremental as it builds on existing annotation methods.
The study tackled inconsistent labeling of offensive language by introducing a prescriptive annotation benchmark, resulting in datasets with higher inter-annotator agreement and showing that smaller models fine-tuned on multi-source LLM-annotated data outperform those trained on larger human-annotated datasets.
This study introduces a prescriptive annotation benchmark grounded in humanities research to ensure consistent, unbiased labeling of offensive language, particularly for casual and non-mainstream language uses. We contribute two newly annotated datasets that achieve higher inter-annotator agreement between human and language model (LLM) annotations compared to original datasets based on descriptive instructions. Our experiments show that LLMs can serve as effective alternatives when professional annotators are unavailable. Moreover, smaller models fine-tuned on multi-source LLM-annotated data outperform models trained on larger, single-source human-annotated datasets. These findings highlight the value of structured guidelines in reducing subjective variability, maintaining performance with limited data, and embracing language diversity. Content Warning: This article only analyzes offensive language for academic purposes. Discretion is advised.