Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
This work addresses the problem of evaluating the utility of visual explanations in AI-assisted decision-making for researchers and practitioners, highlighting incremental insights into explanation effectiveness.
The study conducted a randomized controlled trial on a model-in-the-loop regression task to measure the impact of visual explanations on human accuracy and trust, finding that model predictions improved human accuracy but visual explanations did not significantly affect accuracy or trust regardless of their quality.
We present a randomized controlled trial for a model-in-the-loop regression task, with the goal of measuring the extent to which (1) good explanations of model predictions increase human accuracy, and (2) faulty explanations decrease human trust in the model. We study explanations based on visual saliency in an image-based age prediction task for which humans and learned models are individually capable but not highly proficient and frequently disagree. Our experimental design separates model quality from explanation quality, and makes it possible to compare treatments involving a variety of explanations of varying levels of quality. We find that presenting model predictions improves human accuracy. However, visual explanations of various kinds fail to significantly alter human accuracy or trust in the model - regardless of whether explanations characterize an accurate model, an inaccurate one, or are generated randomly and independently of the input image. These findings suggest the need for greater evaluation of explanations in downstream decision making tasks, better design-based tools for presenting explanations to users, and better approaches for generating explanations.