On Evaluating Explanation Utility for Human-AI Decision Making in NLP
This work addresses the need for better evaluation methods in explainable AI for NLP researchers, but it is incremental as it builds on existing debates and provides guidelines rather than a breakthrough.
The paper tackles the problem of evaluating explanation utility in human-AI decision making in NLP, finding that only 4 out of over 50 datasets meet criteria for application-grounded studies and showing that AI predictions, with or without explanations, do not speed up decision-making without compromising performance.
Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations help people in situations they are introduced for. More human-centered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies. To aid with this, we first review existing metrics suitable for application-grounded evaluation. We then establish criteria to select appropriate datasets, and using them, we find that only 4 out of over 50 datasets available for explainability research in NLP meet them. We then demonstrate the importance of reassessing the state of the art to form and study human-AI teams: teaming people with models for certain tasks might only now start to make sense, and for others, it remains unsound. Finally, we present the exemplar studies of human-AI decision-making for one of the identified tasks -- verifying the correctness of a legal claim given a contract. Our results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance. We argue for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role.