Why Attentions May Not Be Interpretable?
This addresses a foundational issue in interpretability for researchers and practitioners using attention-based models, though it is incremental in proposing mitigations rather than a new paradigm.
The paper tackles the problem that attention weights in neural networks often fail to serve as reliable importance indicators, highlighting less meaningful tokens and showing low correlation with other measures. It identifies combinatorial shortcuts as a root cause and proposes methods that empirically improve interpretability.
Attention-based methods have played important roles in model interpretations, where the calculated attention weights are expected to highlight the critical parts of inputs~(e.g., keywords in sentences). However, recent research found that attention-as-importance interpretations often do not work as we expected. For example, learned attention weights sometimes highlight less meaningful tokens like "[SEP]", ",", and ".", and are frequently uncorrelated with other feature importance indicators like gradient-based measures. A recent debate over whether attention is an explanation or not has drawn considerable interest. In this paper, we demonstrate that one root cause of this phenomenon is the combinatorial shortcuts, which means that, in addition to the highlighted parts, the attention weights themselves may carry extra information that could be utilized by downstream models after attention layers. As a result, the attention weights are no longer pure importance indicators. We theoretically analyze combinatorial shortcuts, design one intuitive experiment to show their existence, and propose two methods to mitigate this issue. We conduct empirical studies on attention-based interpretation models. The results show that the proposed methods can effectively improve the interpretability of attention mechanisms.