Interpretation of the Transformer and Improvement of the Extractor
This work addresses the problem of enhancing Transformer models for researchers and practitioners, but it appears incremental as it builds on existing Extractor methods.
The authors tackled the challenge of improving the Transformer architecture by first providing a comprehensive interpretation of it and the Extractor family, then proposing an improved Extractor that outperforms self-attention without extra parameters, with experimental results showing better performance.
It has been over six years since the Transformer architecture was put forward. Surprisingly, the vanilla Transformer architecture is still widely used today. One reason is that the lack of deep understanding and comprehensive interpretation of the Transformer architecture makes it more challenging to improve the Transformer architecture. In this paper, we first interpret the Transformer architecture comprehensively in plain words based on our understanding and experiences. The interpretations are further proved and verified. These interpretations also cover the Extractor, a family of drop-in replacements for the multi-head self-attention in the Transformer architecture. Then, we propose an improvement on a type of the Extractor that outperforms the self-attention, without introducing additional trainable parameters. Experimental results demonstrate that the improved Extractor performs even better, showing a way to improve the Transformer architecture.