Multi-Perspective Stance Detection
This addresses the issue of diverse human perspectives in AI models for NLP, though it appears incremental as it builds on existing classification methods.
The study tackled the problem of annotator disagreement in subjective NLP tasks by using a multi-perspective approach for stance detection, resulting in better classification performance compared to a baseline using a single label.
Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspective-aware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.