Towards Multi-Stakeholder Evaluation of ML Models: A Crowdsourcing Study on Metric Preferences in Job-matching System
This work addresses the challenge of incorporating diverse stakeholder opinions into metric selection for ML systems, particularly in job-matching, though it is incremental as it builds on existing fairness and evaluation research.
The study tackled the problem of selecting evaluation metrics for ML models by investigating stakeholder preferences through a crowdsourcing experiment with 837 participants in a job-matching system, finding that participants' utility values for seven metrics varied and could be clustered into five groups with distinct tendencies.
While machine learning (ML) technology affects diverse stakeholders, there is no one-size-fits-all metric to evaluate the quality of outputs, including performance and fairness. Using predetermined metrics without soliciting stakeholder opinions is problematic because it leads to an unfair disregard for stakeholders in the ML pipeline. In this study, to establish practical ways to incorporate diverse stakeholder opinions into the selection of metrics for ML, we investigate participants' preferences for different metrics by using crowdsourcing. We ask 837 participants to choose a better model from two hypothetical ML models in a hypothetical job-matching system twenty times and calculate their utility values for seven metrics. To examine the participants' feedback in detail, we divide them into five clusters based on their utility values and analyze the tendencies of each cluster, including their preferences for metrics and common attributes. Based on the results, we discuss the points that should be considered when selecting appropriate metrics and evaluating ML models with multiple stakeholders.