Embracing Diversity: A Multi-Perspective Approach with Soft Labels
This work addresses the need for more responsible AI models in subjective tasks like stance detection by leveraging annotation diversity, though it is incremental as it builds on prior multi-perspective approaches.
The paper tackles stance detection on controversial topics by proposing a multi-perspective framework that incorporates diverse human and LLM annotations, resulting in improved classification performance with higher F1-scores compared to traditional single-ground-truth methods.
Prior studies show that adopting the annotation diversity shaped by different backgrounds and life experiences and incorporating them into the model learning, i.e. multi-perspective approach, contribute to the development of more responsible models. Thus, in this paper we propose a new framework for designing and further evaluating perspective-aware models on stance detection task,in which multiple annotators assign stances based on a controversial topic. We also share a new dataset established through obtaining both human and LLM annotations. Results show that the multi-perspective approach yields better classification performance (higher F1-scores), outperforming the traditional approaches that use a single ground-truth, while displaying lower model confidence scores, probably due to the high level of subjectivity of the stance detection task.