Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
This work addresses the problem of improving Transformer efficiency and performance for researchers and practitioners in NLP, though it is incremental as it builds on existing attention and memory mechanisms.
The paper investigates whether active-memory mechanisms can replace self-attention in Transformers, finding that active-memory alone achieves comparable results for language modeling, but combining both yields optimal performance, with active-memory alone outperforming in some algorithmic tasks.
The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate whether various active-memory mechanisms could replace self-attention in a Transformer. Our experiments suggest that active-memory alone achieves comparable results to the self-attention mechanism for language modelling, but optimal results are mostly achieved by using both active-memory and self-attention mechanisms together. We also note that, for some specific algorithmic tasks, active-memory mechanisms alone outperform both self-attention and a combination of the two.