A Note about: Local Explanation Methods for Deep Neural Networks lack Sensitivity to Parameter Values
This work clarifies a key issue in interpretability for AI researchers, though it is incremental as it corrects prior analysis without introducing new methods.
The paper addresses the claim that local explanation methods for deep neural networks lack sensitivity to parameter values by showing that the original findings were due to methodological choices, and when corrected, integrated gradients attributions for random and learned networks are uncorrelated.
Local explanation methods, also known as attribution methods, attribute a deep network's prediction to its input (cf. Baehrens et al. (2010)). We respond to the claim from Adebayo et al. (2018) that local explanation methods lack sensitivity, i.e., DNNs with randomly-initialized weights produce explanations that are both visually and quantitatively similar to those produced by DNNs with learned weights. Further investigation reveals that their findings are due to two choices in their analysis: (a) ignoring the signs of the attributions; and (b) for integrated gradients (IG), including pixels in their analysis that have zero attributions by choice of the baseline (an auxiliary input relative to which the attributions are computed). When both factors are accounted for, IG attributions for a random network and the actual network are uncorrelated. Our investigation also sheds light on how these issues affect visualizations, although we note that more work is needed to understand how viewers interpret the difference between the random and the actual attributions.