On Explaining with Attention Matrices
This work addresses the interpretability of transformer models for researchers and practitioners, providing a method to validate attention as an explanatory tool, though it is incremental in refining existing debates rather than introducing a new paradigm.
The paper tackles the problem of whether attention weights in transformers have explanatory relevance for model predictions, showing that formal arguments against their relevance are incorrect and introducing 'efficient attention' as a method to isolate effective components, which is demonstrated to have a causal role in NLP tasks requiring contextual information.
This paper explores the much discussed, possible explanatory link between attention weights (AW) in transformer models and predicted output. Contrary to intuition and early research on attention, more recent prior research has provided formal arguments and empirical evidence that AW are not explanatorily relevant. We show that the formal arguments are incorrect. We introduce and effectively compute efficient attention, which isolates the effective components of attention matrices in tasks and models in which AW play an explanatory role. We show that efficient attention has a causal role (provides minimally necessary and sufficient conditions) for predicting model output in NLP tasks requiring contextual information, and we show, contrary to [7], that efficient attention matrices are probability distributions and are effectively calculable. Thus, they should play an important part in the explanation of attention based model behavior. We offer empirical experiments in support of our method illustrating various properties of efficient attention with various metrics on four datasets.