Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture
This provides a foundational mathematical interpretation of Transformer architecture for the machine learning community, though it is incremental as it builds on existing distributional semantics principles.
The paper tackles the problem of interpreting self-attention in Transformers by connecting it to distributional semantics, showing that it emerges from projecting corpus-level co-occurrence statistics into sequence context, with the query-key-value mechanism and other components arising as natural extensions.
This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context. Starting from the co-occurrence matrix underlying GloVe embeddings, we demonstrate how the projection naturally captures contextual influence, with the query-key-value mechanism arising as the natural asymmetric extension for modeling directional relationships. Positional encodings and multi-head attention then follow as structured refinements of this same projection principle. Our analysis demonstrates that the Transformer architecture's particular algebraic form follows from these projection principles rather than being an arbitrary design choice.