Is Attention Interpretation? A Quantitative Assessment On Sets
This addresses the interpretability debate for attention mechanisms in machine learning, particularly for set-based tasks, but is incremental as it builds on existing methods with a quantitative assessment.
The study tackled the problem of whether attention scores can reliably indicate the importance of data components in set machine learning, finding that attention often reflects importance but can fail silently, and proposed ensembling to reduce misleading explanations.
The debate around the interpretability of attention mechanisms is centered on whether attention scores can be used as a proxy for the relative amounts of signal carried by sub-components of data. We propose to study the interpretability of attention in the context of set machine learning, where each data point is composed of an unordered collection of instances with a global label. For classical multiple-instance-learning problems and simple extensions, there is a well-defined "importance" ground truth that can be leveraged to cast interpretation as a binary classification problem, which we can quantitatively evaluate. By building synthetic datasets over several data modalities, we perform a systematic assessment of attention-based interpretations. We find that attention distributions are indeed often reflective of the relative importance of individual instances, but that silent failures happen where a model will have high classification performance but attention patterns that do not align with expectations. Based on these observations, we propose to use ensembling to minimize the risk of misleading attention-based explanations.