Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models
This work addresses the efficiency and design of transformer models for language modeling, offering incremental improvements to a widely used architecture.
The paper tackles the problem of understanding the importance of feedforward networks (FFNs) in transformer models, showing through experiments that FFNs are crucial for performance and that a three-layer FFN configuration with fewer blocks outperforms the standard two-layer setup, achieving lower training loss with fewer parameters and less time.
Decoder-only transformer networks have become incredibly popular for language modeling tasks. State-of-the-art models can have over a hundred transformer blocks, containing billions of trainable parameters, and are trained on trillions of tokens of text. Each transformer block typically consists of a multi-head attention (MHA) mechanism and a two-layer fully connected feedforward network (FFN). In this paper, we examine the importance of the FFN during the model pre-training process through a series of experiments, confirming that the FFN is important to model performance. Furthermore, we show that models using a transformer block configuration with three-layer FFNs with fewer such blocks outperform the standard two-layer configuration delivering lower training loss with fewer total parameters in less time.