Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
This addresses a key bottleneck for researchers and practitioners needing longer context in language models without costly retraining, though it is incremental as it builds on existing positional embedding techniques.
The paper tackles the problem of extending the context length of pretrained language models without expensive finetuning by introducing DroPE, a method that drops positional embeddings after training, enabling zero-shot context extension and outperforming previous methods.
So far, expensive finetuning beyond the pretraining sequence length has been a requirement for effectively extending the context of language models (LM). In this work, we break this key bottleneck by Dropping the Positional Embeddings of LMs after training (DroPE). Our simple method is motivated by three key theoretical and empirical observations. First, positional embeddings (PEs) serve a crucial role during pretraining, providing an important inductive bias that significantly facilitates convergence. Second, over-reliance on this explicit positional information is also precisely what prevents test-time generalization to sequences of unseen length, even when using popular PE-scaling methods. Third, positional embeddings are not an inherent requirement of effective language modeling and can be safely removed after pretraining, following a short recalibration phase. Empirically, DroPE yields seamless zero-shot context extension without any long-context finetuning, quickly adapting pretrained LMs without compromising their capabilities in the original training context. Our findings hold across different models and dataset sizes, far outperforming previous specialized architectures and established rotary positional embedding scaling methods.