Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
This work addresses length extrapolation challenges for language models, offering a novel positional encoding method that enhances performance on long sequences, though it is incremental relative to existing positional encoding techniques.
The paper tackles the problem of length extrapolation in language models by introducing Bilevel Positional Encoding (BiPE), which blends intra-segment and inter-segment encodings to improve positional information handling, resulting in superior extrapolation capabilities across diverse text tasks.
In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.