Improving Model Understanding and Trust with Counterfactual Explanations of Model Confidence
This addresses the issue of building trust in human-AI interaction systems, though it is incremental as it builds on existing confidence score communication methods.
The paper tackled the problem of explaining why an AI model is confident in its predictions, and found that counterfactual explanations of confidence scores help users better understand and trust the model in human-subject studies.
In this paper, we show that counterfactual explanations of confidence scores help users better understand and better trust an AI model's prediction in human-subject studies. Showing confidence scores in human-agent interaction systems can help build trust between humans and AI systems. However, most existing research only used the confidence score as a form of communication, and we still lack ways to explain why the algorithm is confident. This paper also presents two methods for understanding model confidence using counterfactual explanation: (1) based on counterfactual examples; and (2) based on visualisation of the counterfactual space.