SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
This work addresses the challenge of scaling language models to longer contexts more efficiently, which is crucial for applications like document analysis and extended conversations, though it appears incremental as it builds on existing Transformer and attention mechanisms.
The authors tackled the problem of enabling language models to handle longer sequences than seen during training by introducing SWAN-GPT, a decoder-only Transformer architecture that robustly generalizes to extended contexts without additional long-context training, achieving strong performance and improved computational efficiency.
We present a decoder-only Transformer architecture that robustly generalizes to sequence lengths substantially longer than those seen during training. Our model, SWAN-GPT, interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positional encodings (SWA-RoPE). Experiments demonstrate strong performance on sequence lengths significantly longer than the training length without the need for additional long-context training. This robust length extrapolation is achieved through our novel architecture, enhanced by a straightforward dynamic scaling of attention scores during inference. In addition, SWAN-GPT is more computationally efficient than standard GPT architectures, resulting in cheaper training and higher throughput. Further, we demonstrate that existing pre-trained decoder-only models can be efficiently converted to the SWAN architecture with minimal continued training, enabling longer contexts. Overall, our work presents an effective approach for scaling language models to longer contexts in a robust and efficient manner.