A Multiscale Visualization of Attention in the Transformer Model
This tool aids researchers and practitioners in understanding and debugging Transformer models, though it is incremental as it builds on existing attention visualization methods.
The authors tackled the challenge of interpreting the multi-layer, multi-head attention mechanism in Transformer models by developing an open-source visualization tool that provides multiscale insights, demonstrated on BERT and GPT-2 with use cases like detecting bias and linking neurons to behavior.
The Transformer is a sequence model that forgoes traditional recurrent architectures in favor of a fully attention-based approach. Besides improving performance, an advantage of using attention is that it can also help to interpret a model by showing how the model assigns weight to different input elements. However, the multi-layer, multi-head attention mechanism in the Transformer model can be difficult to decipher. To make the model more accessible, we introduce an open-source tool that visualizes attention at multiple scales, each of which provides a unique perspective on the attention mechanism. We demonstrate the tool on BERT and OpenAI GPT-2 and present three example use cases: detecting model bias, locating relevant attention heads, and linking neurons to model behavior.