Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science
This work addresses the need for greater transparency and ethical awareness in data science practices, though it is incremental in adapting existing qualitative methods to computational contexts.
The paper tackles the problem of unrecognized discretionary choices in data science by adapting qualitative research concepts of positionality and reflexivity to create a framework for understanding and disclosing subjectivity in model development, using a case study on toxic comment classifiers.
Data science and machine learning provide indispensable techniques for understanding phenomena at scale, but the discretionary choices made when doing this work are often not recognized. Drawing from qualitative research practices, we describe how the concepts of positionality and reflexivity can be adapted to provide a framework for understanding, discussing, and disclosing the discretionary choices and subjectivity inherent to data science work. We first introduce the concepts of model positionality and computational reflexivity that can help data scientists to reflect on and communicate the social and cultural context of a model's development and use, the data annotators and their annotations, and the data scientists themselves. We then describe the unique challenges of adapting these concepts for data science work and offer annotator fingerprinting and position mining as promising solutions. Finally, we demonstrate these techniques in a case study of the development of classifiers for toxic commenting in online communities.