Length Generalization of Causal Transformers without Position Encoding
This work addresses the challenge of generalizing to longer sequences for language models, which is crucial for real-world applications, but it is incremental as it builds on existing NoPE methods.
The paper tackled the problem of length generalization in Transformers without position encodings (NoPE), finding that they have limited context length due to attention distraction, and proposed a parameter-efficient tuning method that substantially expands context size, achieving competitive performance on tasks like long sequence language modeling and passkey retrieval.
Generalizing to longer sentences is important for recent Transformer-based language models. Besides algorithms manipulating explicit position features, the success of Transformers without position encodings (NoPE) provides a new way to overcome the challenge. In this paper, we study the length generalization property of NoPE. We find that although NoPE can extend to longer sequences than the commonly used explicit position encodings, it still has a limited context length. We identify a connection between the failure of NoPE's generalization and the distraction of attention distributions. We propose a parameter-efficient tuning for searching attention heads' best temperature hyper-parameters, which substantially expands NoPE's context size. Experiments on long sequence language modeling, the synthetic passkey retrieval task and real-world long context tasks show that NoPE can achieve competitive performances with state-of-the-art length generalization algorithms. The source code is publicly accessible