Position Interpolation

“The two training-free length extrapolation baselines, Dual Chunk Attention and Positional Interpolation, shown in Table~tab:qwen_math_extrapolation, achieve accuracies close to zero, demonstrating astonishingly poor performance.”

“models trained in this manner adapt to long context lengths very slowly”

“PI scales the positions of long texts that exceed the context window down to the original window size. However, it compresses distances between nearby tokens, which can degrade performance.”

“While large scale content is properly synthesized in this approach, the missing high-frequencies manifest as blurriness and lack of fine detail”

“However, as the interpolation factor increases, PI experiences a substantial decline in positional resolution among tokens, detrimentally affecting long-context modeling performance.”

“However, positional embedding exhibits complex non-uniform information entropy in the Transformer architecture. Such subtle non-uniformity is not effectively leveraged by existing approaches, leading to information loss and hence limiting the context window size.”

“both methods introduce substantial time and memory overhead”

“Previous approaches, such as PI and NTK, aim to mitigate this issue by reducing the magnitude of relative position information. Consequently, the relative position matrix is also scaled: leading to inferior resolution of the position information and weak extrapolation ability.”