D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
This addresses the need for culturally sensitive NLP models by providing a dataset that captures diverse values beyond Western contexts, though it is incremental in expanding existing work on annotator subjectivity.
The paper tackles the problem of annotator subjectivity in offensive language detection by introducing D3CODE, a large-scale cross-cultural dataset with over 4.5K sentences annotated by over 4k annotators from 21 countries, revealing substantial regional variations influenced by individual moral values.
While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection. Recent studies that have critically examined this issue are often situated in the Western context, and solely document differences across age, gender, or racial groups. As a result, NLP research on subjectivity have overlooked the fact that individuals within demographic groups may hold diverse values, which can influence their perceptions beyond their group norms. To effectively incorporate these considerations into NLP pipelines, we need datasets with extensive parallel annotations from various social and cultural groups. In this paper we introduce the \dataset dataset: a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K sentences annotated by a pool of over 4k annotators, balanced across gender and age, from across 21 countries, representing eight geo-cultural regions. The dataset contains annotators' moral values captured along six moral foundations: care, equality, proportionality, authority, loyalty, and purity. Our analyses reveal substantial regional variations in annotators' perceptions that are shaped by individual moral values, offering crucial insights for building pluralistic, culturally sensitive NLP models.