Transformer on a Diet
This work addresses the computational inefficiency of Transformers for practitioners, though it is incremental as it builds on existing architectures.
The paper tackles the problem of heavy Transformer architectures by exploring three light Transformer designs, achieving up to 70% parameter reduction while maintaining competitive perplexity on language model benchmarks.
Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. However, recent developments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effectiveness. In this paper, we explore three carefully-designed light Transformer architectures to figure out whether the Transformer with less computations could produce competitive results. Experimental results on language model benchmark datasets hint that such trade-off is promising, and the light Transformer reduces 70% parameters at best, while obtains competitive perplexity compared to standard Transformer. The source code is publicly available.