VOICE: Variance of Induced Contrastive Explanations to quantify Uncertainty in Neural Network Interpretability
This work addresses the reliability of interpretability methods for neural networks, which is crucial for users in fields like healthcare and autonomous systems, though it is incremental in extending existing evaluation strategies.
The paper tackles the problem of quantifying predictive uncertainty in gradient-based visual explanations for neural networks, showing that each image, network, and explanation has unique uncertainty, with key findings including reduced trustworthiness in incorrect predictions and empirical similarity to epistemic uncertainty.
In this paper, we visualize and quantify the predictive uncertainty of gradient-based post hoc visual explanations for neural networks. Predictive uncertainty refers to the variability in the network predictions under perturbations to the input. Visual post hoc explainability techniques highlight features within an image to justify a network's prediction. We theoretically show that existing evaluation strategies of visual explanatory techniques partially reduce the predictive uncertainty of neural networks. This analysis allows us to construct a plug in approach to visualize and quantify the remaining predictive uncertainty of any gradient-based explanatory technique. We show that every image, network, prediction, and explanatory technique has a unique uncertainty. The proposed uncertainty visualization and quantification yields two key observations. Firstly, oftentimes under incorrect predictions, explanatory techniques are uncertain about the same features that they are attributing the predictions to, thereby reducing the trustworthiness of the explanation. Secondly, objective metrics of an explanation's uncertainty, empirically behave similarly to epistemic uncertainty. We support these observations on two datasets, four explanatory techniques, and six neural network architectures. The code is available at https://github.com/olivesgatech/VOICE-Uncertainty.