AttentionViz: A Global View of Transformer Attention
This work addresses the challenge of interpretability for researchers working with transformer models, though it is incremental as it builds on existing visualization methods.
The paper tackles the problem of understanding the inner workings of transformer models by introducing a new visualization technique that analyzes global patterns in self-attention across multiple input sequences, resulting in an interactive tool that provides insights into query-key interactions in language and vision transformers.
Transformer models are revolutionizing machine learning, but their inner workings remain mysterious. In this work, we present a new visualization technique designed to help researchers understand the self-attention mechanism in transformers that allows these models to learn rich, contextual relationships between elements of a sequence. The main idea behind our method is to visualize a joint embedding of the query and key vectors used by transformer models to compute attention. Unlike previous attention visualization techniques, our approach enables the analysis of global patterns across multiple input sequences. We create an interactive visualization tool, AttentionViz (demo: http://attentionviz.com), based on these joint query-key embeddings, and use it to study attention mechanisms in both language and vision transformers. We demonstrate the utility of our approach in improving model understanding and offering new insights about query-key interactions through several application scenarios and expert feedback.