Decision Explanation and Feature Importance for Invertible Networks
This addresses the interpretability issue in deep learning for researchers and practitioners, offering a novel approach but is incremental in building on invertible network concepts.
The paper tackles the problem of interpreting deep neural networks by leveraging invertible networks to reconstruct inputs from outputs, enabling the inversion of decision boundaries from feature to input space and defining explanations as differences between data points and their projections onto the decision boundary. It results in a method that provides decision explanations and feature importance, with implementation available online.
Deep neural networks are vulnerable to adversarial attacks and hard to interpret because of their black-box nature. The recently proposed invertible network is able to accurately reconstruct the inputs to a layer from its outputs, thus has the potential to unravel the black-box model. An invertible network classifier can be viewed as a two-stage model: (1) invertible transformation from input space to the feature space; (2) a linear classifier in the feature space. We can determine the decision boundary of a linear classifier in the feature space; since the transform is invertible, we can invert the decision boundary from the feature space to the input space. Furthermore, we propose to determine the projection of a data point onto the decision boundary, and define explanation as the difference between data and its projection. Finally, we propose to locally approximate a neural network with its first-order Taylor expansion, and define feature importance using a local linear model. We provide the implementation of our method: \url{https://github.com/juntang-zhuang/explain_invertible}.