Long-context / context-window extension
Position Interpolation
Extending Context Window of Large Language Models via Positional Interpolation
Heavily superseded — a standard baseline that newer methods routinely beat
8 papers critique it · 5 beat it on benchmarks
What papers say
Verbatim critique sentences, each from a paper that cites Position Interpolation as a baseline.
The two training-free length extrapolation baselines, Dual Chunk Attention and Positional Interpolation, shown in Table~tab:qwen_math_extrapolation, achieve accuracies close to zero, demonstrating astonishingly poor performance.
“models trained in this manner adapt to long context lengths very slowly”
“PI scales the positions of long texts that exceed the context window down to the original window size. However, it compresses distances between nearby tokens, which can degrade performance.”
“While large scale content is properly synthesized in this approach, the missing high-frequencies manifest as blurriness and lack of fine detail”
“However, as the interpolation factor increases, PI experiences a substantial decline in positional resolution among tokens, detrimentally affecting long-context modeling performance.”
“However, positional embedding exhibits complex non-uniform information entropy in the Transformer architecture. Such subtle non-uniformity is not effectively leveraged by existing approaches, leading to information loss and hence limiting the context window size.”
“both methods introduce substantial time and memory overhead”
“Previous approaches, such as PI and NTK, aim to mitigate this issue by reducing the magnitude of relative position information. Consequently, the relative position matrix is also scaled: leading to inferior resolution of the position information and weak extrapolation ability.”
Beaten on benchmarks
Head-to-head results where a newer method reports beating Position Interpolation. Values are copied from the source paper's tables — verify against the cited paper.
DoPE-by-Gaussian beats Position Interpolation
70.083 vs 11.771
Original (64k) · [64k tokens]
DoPE: Denoising Rotary Position EmbeddingDCIS beats Position Interpolation
2.73 vs 55.97
PPL · [Fine-tuning Llama2-7B, evaluation at target length 64k]
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
What to use instead
Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.