(Re)Defining Expertise in Machine Learning Development
This addresses a conceptual gap for ML researchers and practitioners by clarifying expert engagement, but it is incremental as it synthesizes existing literature without new empirical data.
The paper tackles the problem of undefined expertise in machine learning development by conducting a systematic literature review to analyze how expertise is defined and the roles experts play, resulting in a taxonomy to highlight limits and opportunities.
Domain experts are often engaged in the development of machine learning systems in a variety of ways, such as in data collection and evaluation of system performance. At the same time, who counts as an 'expert' and what constitutes 'expertise' is not always explicitly defined. In this project, we conduct a systematic literature review of machine learning research to understand 1) the bases on which expertise is defined and recognized and 2) the roles experts play in ML development. Our goal is to produce a high-level taxonomy to highlight limits and opportunities in how experts are identified and engaged in ML research.