Fidelity of Interpretability Methods and Perturbation Artifacts in Neural Networks
This addresses the problem of unreliable evaluation of interpretability methods for researchers and practitioners, but it is incremental as it builds on existing perturbation-based approaches.
The paper tackles the challenge of evaluating post-hoc interpretability methods in deep neural networks by proposing a method to estimate the impact of perturbation artifacts on fidelity estimation, demonstrating it on ResNet-50 trained on ImageNet with four popular methods.
Despite excellent performance of deep neural networks (DNNs) in image classification, detection, and prediction, characterizing how DNNs make a given decision remains an open problem, resulting in a number of interpretability methods. Post-hoc interpretability methods primarily aim to quantify the importance of input features with respect to the class probabilities. However, due to the lack of ground truth and the existence of interpretability methods with diverse operating characteristics, evaluating these methods is a crucial challenge. A popular approach to evaluate interpretability methods is to perturb input features deemed important for a given prediction and observe the decrease in accuracy. However, perturbation itself may introduce artifacts. We propose a method for estimating the impact of such artifacts on the fidelity estimation by utilizing model accuracy curves from perturbing input features according to the Most Import First (MIF) and Least Import First (LIF) orders. Using the ResNet-50 trained on the ImageNet, we demonstrate the proposed fidelity estimation of four popular post-hoc interpretability methods.