Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems
This addresses the challenge of aligning algorithmic tools with community values for Wikipedia stakeholders, but is incremental as it focuses on understanding rather than solving these tensions.
The study tackled the problem of machine learning-based algorithms like ORES on Wikipedia potentially conflicting with community values, and identified five key stakeholder values such as reducing maintenance effort and maintaining human authority, revealing tensions among them.
On Wikipedia, sophisticated algorithmic tools are used to assess the quality of edits and take corrective actions. However, algorithms can fail to solve the problems they were designed for if they conflict with the values of communities who use them. In this study, we take a Value-Sensitive Algorithm Design approach to understanding a community-created and -maintained machine learning-based algorithm called the Objective Revision Evaluation System (ORES)---a quality prediction system used in numerous Wikipedia applications and contexts. Five major values converged across stakeholder groups that ORES (and its dependent applications) should: (1) reduce the effort of community maintenance, (2) maintain human judgement as the final authority, (3) support differing peoples' differing workflows, (4) encourage positive engagement with diverse editor groups, and (5) establish trustworthiness of people and algorithms within the community. We reveal tensions between these values and discuss implications for future research to improve algorithms like ORES.