Do Explanations Explain? Model Knows Best
This work addresses the challenge of trust in explanation methods for neural networks, providing a tool for researchers to empirically evaluate and compare feature attribution techniques, though it is incremental as it builds on existing axiomatic approaches.
The authors tackled the problem of inconsistent feature attributions in neural network explanations by proposing an empirical framework that uses the model itself to generate controlled inputs for evaluating explanation methods against axioms. They applied this framework to assess several existing explanation solutions, revealing their properties and drawbacks.
It is a mystery which input features contribute to a neural network's output. Various explanation (feature attribution) methods are proposed in the literature to shed light on the problem. One peculiar observation is that these explanations (attributions) point to different features as being important. The phenomenon raises the question, which explanation to trust? We propose a framework for evaluating the explanations using the neural network model itself. The framework leverages the network to generate input features that impose a particular behavior on the output. Using the generated features, we devise controlled experimental setups to evaluate whether an explanation method conforms to an axiom. Thus we propose an empirical framework for axiomatic evaluation of explanation methods. We evaluate well-known and promising explanation solutions using the proposed framework. The framework provides a toolset to reveal properties and drawbacks within existing and future explanation solutions.