Generalizing Backpropagation for Gradient-Based Interpretability
This provides a more detailed interpretability method for researchers and practitioners working with complex models like BERT, though it is incremental as it builds on existing gradient-based approaches.
The paper tackles the problem of interpreting deep neural networks by generalizing backpropagation using semirings to compute interpretable statistics like highest-weighted paths and entropy, applying it to validate gradient flow importance and identify key pathways in BERT for subject-verb agreement tasks.
Many popular feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model's output with respect to its inputs. While these methods can indicate which input features may be important for the model's prediction, they reveal little about the inner workings of the model itself. In this paper, we observe that the gradient computation of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy. We implement this generalized algorithm, evaluate it on synthetic datasets to better understand the statistics it computes, and apply it to study BERT's behavior on the subject-verb number agreement task (SVA). With this method, we (a) validate that the amount of gradient flow through a component of a model reflects its importance to a prediction and (b) for SVA, identify which pathways of the self-attention mechanism are most important.