Learning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts
For NLP researchers working on subjective tasks like moral classification, this work highlights the importance of modeling individual annotator perspectives rather than relying on aggregated labels.
The paper addresses the problem of subjectivity in moral classification of texts by proposing a model that learns annotator-specific features, improving predictions of individual annotations and revealing insights into moral perspectives. The model shows that aggregated labels hide variation and give misleading performance impressions.
Understanding moral values in social media text offers insight into moral judgement formation, and supervised NLP models trained on crowdsourced data have achieved strong classification performance. However, most approaches simplify the problem by aggregating multiple annotators' labels into a single "ground truth", overlooking the inherent subjectivity of the task. In practice, there are disagreements between annotators caused by personal viewpoint or inherent ambiguities, particularly for short tweets. Here, we extend a pretrained language model with a layer that learns annotator-specific features. Our model improves predictions of individual annotations and yields representations that reveal meaningful insights into annotators' moral perspectives. We show that models trained on aggregated labels may hide variation and give a misleading impression of performance. Overall, we demonstrate that disagreement reflects the inherent subjectivity of the task and that modelling individual perspectives creates benefits for moral classification of texts.