Improving Deep Learning Interpretability by Saliency Guided Training
This addresses the issue of unreliable interpretability in deep learning models for researchers and practitioners, though it is incremental as it builds on existing saliency methods.
The paper tackles the problem of noisy gradients causing unfaithful feature attributions in saliency methods by introducing a saliency guided training procedure that masks features with small gradients while maximizing output similarity, and shows it significantly improves model interpretability across domains while preserving predictive performance.
Saliency methods have been widely used to highlight important input features in model predictions. Most existing methods use backpropagation on a modified gradient function to generate saliency maps. Thus, noisy gradients can result in unfaithful feature attributions. In this paper, we tackle this issue and introduce a {\it saliency guided training}procedure for neural networks to reduce noisy gradients used in predictions while retaining the predictive performance of the model. Our saliency guided training procedure iteratively masks features with small and potentially noisy gradients while maximizing the similarity of model outputs for both masked and unmasked inputs. We apply the saliency guided training procedure to various synthetic and real data sets from computer vision, natural language processing, and time series across diverse neural architectures, including Recurrent Neural Networks, Convolutional Networks, and Transformers. Through qualitative and quantitative evaluations, we show that saliency guided training procedure significantly improves model interpretability across various domains while preserving its predictive performance.