What Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations
This addresses the need for user-centered explainable AI by identifying what questions people actually want answered about ML models, though it is incremental as it focuses on a specific case study.
The researchers tackled the problem that explainable AI methods are developer-driven rather than user-needs-driven by creating a conversational chatbot called dr_ant to collect what questions human operators ask about a Titanic survival prediction model, resulting in a corpus of 1000+ dialogues analyzed for common question types.
Recently we see a rising number of methods in the field of eXplainable Artificial Intelligence. To our surprise, their development is driven by model developers rather than a study of needs for human end users. The analysis of needs, if done, takes the form of an A/B test rather than a study of open questions. To answer the question "What would a human operator like to ask the ML model?" we propose a conversational system explaining decisions of the predictive model. In this experiment, we developed a chatbot called dr_ant to talk about machine learning model trained to predict survival odds on Titanic. People can talk with dr_ant about different aspects of the model to understand the rationale behind its predictions. Having collected a corpus of 1000+ dialogues, we analyse the most common types of questions that users would like to ask. To our knowledge, it is the first study which uses a conversational system to collect the needs of human operators from the interactive and iterative dialogue explorations of a predictive model.