Neural Response Interpretation through the Lens of Critical Pathways
This work addresses the challenge of interpretability in neural networks for researchers and practitioners, offering a novel method to improve feature attribution, though it is incremental as it builds on prior pruning approaches.
The authors tackled the problem of identifying critical pathways in neural networks for interpreting responses, showing that pruning-based pathways fail to encode critical input information, and proposed a neuron contribution method that achieves locally linear pathways and validates as corresponding to critical features.
Is critical input information encoded in specific sparse pathways within the neural network? In this work, we discuss the problem of identifying these critical pathways and subsequently leverage them for interpreting the network's response to an input. The pruning objective -- selecting the smallest group of neurons for which the response remains equivalent to the original network -- has been previously proposed for identifying critical pathways. We demonstrate that sparse pathways derived from pruning do not necessarily encode critical input information. To ensure sparse pathways include critical fragments of the encoded input information, we propose pathway selection via neurons' contribution to the response. We proceed to explain how critical pathways can reveal critical input features. We prove that pathways selected via neuron contribution are locally linear (in an L2-ball), a property that we use for proposing a feature attribution method: "pathway gradient". We validate our interpretation method using mainstream evaluation experiments. The validation of pathway gradient interpretation method further confirms that selected pathways using neuron contributions correspond to critical input features. The code is publicly available.