Rethinking Positive Aggregation and Propagation of Gradients in Gradient-based Saliency Methods
This work identifies a critical flaw in widely used gradient-based saliency methods, impacting researchers and practitioners who rely on these methods for model interpretability.
This paper empirically demonstrates that two common approaches in gradient-based saliency methods, positive aggregation and positive propagation, compromise their ability to explain model predictions. The authors show that saliency maps generated using these techniques become insensitive to the predicted output and model parameter randomization, particularly for methods like GradCAM++ and LRP.
Saliency methods interpret the prediction of a neural network by showing the importance of input elements for that prediction. A popular family of saliency methods utilize gradient information. In this work, we empirically show that two approaches for handling the gradient information, namely positive aggregation, and positive propagation, break these methods. Though these methods reflect visually salient information in the input, they do not explain the model prediction anymore as the generated saliency maps are insensitive to the predicted output and are insensitive to model parameter randomization. Specifically for methods that aggregate the gradients of a chosen layer such as GradCAM++ and FullGrad, exclusively aggregating positive gradients is detrimental. We further support this by proposing several variants of aggregation methods with positive handling of gradient information. For methods that backpropagate gradient information such as LRP, RectGrad, and Guided Backpropagation, we show the destructive effect of exclusively propagating positive gradient information.