Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
This addresses the gap between fairness toolkit development and real-world usage for ML practitioners, but it is incremental as it builds on existing research without introducing new methods.
The paper tackled the problem of understanding how machine learning practitioners use fairness toolkits in practice, finding that current toolkits often fail to meet practitioner needs and identifying opportunities for improvement.
Recent years have seen the development of many open-source ML fairness toolkits aimed at helping ML practitioners assess and address unfairness in their systems. However, there has been little research investigating how ML practitioners actually use these toolkits in practice. In this paper, we conducted the first in-depth empirical exploration of how industry practitioners (try to) work with existing fairness toolkits. In particular, we conducted think-aloud interviews to understand how participants learn about and use fairness toolkits, and explored the generality of our findings through an anonymous online survey. We identified several opportunities for fairness toolkits to better address practitioner needs and scaffold them in using toolkits effectively and responsibly. Based on these findings, we highlight implications for the design of future open-source fairness toolkits that can support practitioners in better contextualizing, communicating, and collaborating around ML fairness efforts.