Synthesizer: Rethinking Self-Attention in Transformer Models
This work challenges a core assumption in Transformer models, potentially enabling more efficient architectures for NLP tasks, though it appears incremental as it builds on existing attention mechanisms.
The paper investigates the necessity of dot product self-attention in Transformers, finding that random alignment matrices perform competitively and token-token interactions are less critical than assumed, and proposes Synthesizer models that learn synthetic attention weights without such interactions, achieving competitive or superior performance across tasks like machine translation and GLUE benchmarks, with a Random Synthesizer being 60% faster and improving perplexity by 3.5%.
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.