Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers
This work addresses the quadratic complexity bottleneck in transformers for NLP and other domains, offering an incremental improvement over prior FNet models.
The authors tackled the computational inefficiency of transformer encoders by proposing Fast-FNet, which uses efficient Fourier Transform layers to replace attention, achieving competitive performance with fewer parameters, faster training, and less memory usage.
Transformer-based language models utilize the attention mechanism for substantial performance improvements in almost all natural language processing (NLP) tasks. Similar attention structures are also extensively studied in several other areas. Although the attention mechanism enhances the model performances significantly, its quadratic complexity prevents efficient processing of long sequences. Recent works focused on eliminating the disadvantages of computational inefficiency and showed that transformer-based models can still reach competitive results without the attention layer. A pioneering study proposed the FNet, which replaces the attention layer with the Fourier Transform (FT) in the transformer encoder architecture. FNet achieves competitive performances concerning the original transformer encoder model while accelerating training process by removing the computational burden of the attention mechanism. However, the FNet model ignores essential properties of the FT from the classical signal processing that can be leveraged to increase model efficiency further. We propose different methods to deploy FT efficiently in transformer encoder models. Our proposed architectures have smaller number of model parameters, shorter training times, less memory usage, and some additional performance improvements. We demonstrate these improvements through extensive experiments on common benchmarks.