What Makes for a Good Saliency Map? Comparing Strategies for Evaluating Saliency Maps in Explainable AI (XAI)
This work addresses the challenge of how to effectively evaluate saliency maps for researchers and practitioners in explainable AI, providing insights into the discrepancies between different evaluation methods, but it is incremental as it builds on existing evaluation frameworks without introducing new paradigms.
The paper tackled the problem of evaluating saliency maps in explainable AI by comparing three popular methods (LIME, Grad-CAM, Guided Backpropagation) across subjective, objective, and mathematical evaluation families in a user study with 166 participants, finding that these evaluation strategies do not agree in their assessments, with Grad-CAM improving user abilities best and Guided Backpropagation having the most favorable mathematical metrics.
Saliency maps are a popular approach for explaining classifications of (convolutional) neural networks. However, it remains an open question as to how best to evaluate salience maps, with three families of evaluation methods commonly being used: subjective user measures, objective user measures, and mathematical metrics. We examine three of the most popular saliency map approaches (viz., LIME, Grad-CAM, and Guided Backpropagation) in a between subject study (N=166) across these families of evaluation methods. We test 1) for subjective measures, if the maps differ with respect to user trust and satisfaction; 2) for objective measures, if the maps increase users' abilities and thus understanding of a model; 3) for mathematical metrics, which map achieves the best ratings across metrics; and 4) whether the mathematical metrics can be associated with objective user measures. To our knowledge, our study is the first to compare several salience maps across all these evaluation methods$-$with the finding that they do not agree in their assessment (i.e., there was no difference concerning trust and satisfaction, Grad-CAM improved users' abilities best, and Guided Backpropagation had the most favorable mathematical metrics). Additionally, we show that some mathematical metrics were associated with user understanding, although this relationship was often counterintuitive. We discuss these findings in light of general debates concerning the complementary use of user studies and mathematical metrics in the evaluation of explainable AI (XAI) approaches.