Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
This proposes a foundational shift in how ground truth is established in AI, potentially affecting all supervised ML applications by addressing annotation biases.
The paper tackles the problem of reliance on majority-vote annotations in supervised machine learning, advocating for a 'data perspectivism' paradigm that integrates diverse human perspectives to improve knowledge representation, particularly in both subjective and objective tasks like language and medical decision-making.
Most Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. In this article we describe and advocate for a different paradigm, which we call data perspectivism, which moves away from traditional gold standard datasets, towards the adoption of methods that integrate the opinions and perspectives of the human subjects involved in the knowledge representation step of ML processes. Drawing on previous works which inspired our proposal we describe the potential of our proposal for not only the more subjective tasks (e.g. those related to human language) but also to tasks commonly understood as objective (e.g. medical decision making), and present the main advantages of adopting a perspectivist stance in ML, as well as possible disadvantages, and various ways in which such a stance can be implemented in practice. Finally, we share a set of recommendations and outline a research agenda to advance the perspectivist stance in ML.