Explaining Neural Networks with Reasons
This addresses the need for interpretability in AI, particularly for understanding neural network decisions across various architectures and modalities, though it appears incremental as it builds on existing interpretability concepts with a new theoretical framework.
The authors tackled the problem of interpreting neural networks by proposing a method based on a novel theory of reasons, which computes reason vectors for neurons to explain their roles in predictions, showing it is grounded, uniform, scalable, faithful, correct, trainable, and useful for improving robustness and fairness.
We propose a new interpretability method for neural networks, which is based on a novel mathematico-philosophical theory of reasons. Our method computes a vector for each neuron, called its reasons vector. We then can compute how strongly this reasons vector speaks for various propositions, e.g., the proposition that the input image depicts digit 2 or that the input prompt has a negative sentiment. This yields an interpretation of neurons, and groups thereof, that combines a logical and a Bayesian perspective, and accounts for polysemanticity (i.e., that a single neuron can figure in multiple concepts). We show, both theoretically and empirically, that this method is: (1) grounded in a philosophically established notion of explanation, (2) uniform, i.e., applies to the common neural network architectures and modalities, (3) scalable, since computing reason vectors only involves forward-passes in the neural network, (4) faithful, i.e., intervening on a neuron based on its reason vector leads to expected changes in model output, (5) correct in that the model's reasons structure matches that of the data source, (6) trainable, i.e., neural networks can be trained to improve their reason strengths, (7) useful, i.e., it delivers on the needs for interpretability by increasing, e.g., robustness and fairness.