Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks
This work addresses the need for better interpretability in AI systems, particularly for researchers and practitioners using DNNs, though it is incremental as it builds on prior attribution methods.
The paper tackles the problem of interpreting deep neural networks by proposing Relative Attributing Propagation (RAP), which separates positive and negative attributions to neurons, resulting in clearer visualizations and outperforming existing methods across multiple evaluation metrics.
As Deep Neural Networks (DNNs) have demonstrated superhuman performance in a variety of fields, there is an increasing interest in understanding the complex internal mechanisms of DNNs. In this paper, we propose Relative Attributing Propagation (RAP), which decomposes the output predictions of DNNs with a new perspective of separating the relevant (positive) and irrelevant (negative) attributions according to the relative influence between the layers. The relevance of each neuron is identified with respect to its degree of contribution, separated into positive and negative, while preserving the conservation rule. Considering the relevance assigned to neurons in terms of relative priority, RAP allows each neuron to be assigned with a bi-polar importance score concerning the output: from highly relevant to highly irrelevant. Therefore, our method makes it possible to interpret DNNs with much clearer and attentive visualizations of the separated attributions than the conventional explaining methods. To verify that the attributions propagated by RAP correctly account for each meaning, we utilize the evaluation metrics: (i) Outside-inside relevance ratio, (ii) Segmentation mIOU and (iii) Region perturbation. In all experiments and metrics, we present a sizable gap in comparison to the existing literature. Our source code is available in \url{https://github.com/wjNam/Relative_Attributing_Propagation}.