Is Self-Extend superseded?

Self-Extend (Long-context / context-window extension): superseded — cited as a baseline and beaten by newer methods. 5 paper(s) critique it, 4 beat it on benchmarks — #8 of 53 most-superseded. Sub-problem: cluster led by YaRN. Newer alternatives in the same sub-problem include Cross-Resolution Phase-Aligned Attention (CRPA), DoPE, DyPE.

Long-context / context-window extension

Self-Extend

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

Superseded baseline#8 of 53 most-superseded · first seen Jan 2, 2024

Superseded — cited as a baseline and beaten by newer methods

5 papers critique it · 4 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Self-Extend as a baseline.

approaches like ReRoPE rerope2023 and Self-Extend jin2024llm extend sequence lengths by compressing sequence indices, although they necessitate double attention computations, raising computational demands and limiting extrapolation potential.

— DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search

“However, they blindly manipulate the position embeddings equally on all RoPE dimensions without considering RoPE's rotational properties.”

— Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation

“SelfExtend and ChunkLlama inherently disrupt local positional relationships, compromising model performance”

— A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

“SelfExtend's results on Llama3 in LongBench show that it impairs performance within the original window, which also highlights the limitations of manually setting group size.”

— LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training

“While LongLM's method shows promising results on long-context tasks, we propose a more adaptive strategy grounded in the observation that, in natural language, the relevance of a token typically decreases with its distance from the current context.”

— SELF: Self-Extend the Context Length With Logistic Growth Function

Beaten on benchmarks

Head-to-head results where a newer method reports beating Self-Extend. Values are copied from the source paper's tables — verify against the cited paper.

TokenSelect beats Self-Extend
36.77 vs 1.01
Avg. · [Yi-1.5-6B]
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
LaMPE beats Self-Extend
59.48 vs 35.97
Avg. · [128K extrapolation]
LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.