How to Dissect a Muppet: The Structure of Transformer Embedding Spaces
This provides a method for dissecting Transformer embeddings to improve interpretability and efficiency in NLP, though it is incremental as it builds on existing studies.
The paper tackles the problem of understanding Transformer embedding spaces by reframing them as a sum of vector factors, enabling analysis of component impacts and revealing that multi-head attentions and feed-forwards vary in usefulness across downstream applications, with quantitative insights into finetuning effects.
Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm. We show that they can mathematically be reframed as a sum of vector factors and showcase how to use this reframing to study the impact of each component. We provide evidence that multi-head attentions and feed-forwards are not equally useful in all downstream applications, as well as a quantitative overview of the effects of finetuning on the overall embedding space. This approach allows us to draw connections to a wide range of previous studies, from vector space anisotropy to attention weights.