ALiBi (Long-context / context-window extension): superseded — cited as a baseline and beaten by newer methods. 4 paper(s) critique it, 5 beat it on benchmarks — #7 of 53 most-superseded. Sub-problem: cluster led by RoPE. Newer alternatives in the same sub-problem include Mask Prior Suppression and Monotonic RoPE Scaling, CRoPE, C^2RoPE, Imaginary Extension of Rotary Position Embeddings, Selective RoPE.

Long-context / context-window extension

ALiBi

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Superseded baseline#7 of 53 most-superseded · first seen Aug 27, 2021

Superseded — cited as a baseline and beaten by newer methods

4 papers critique it · 5 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites ALiBi as a baseline.

Alibi press2022trainshorttestlong enhanced extrapolation capability through a distance-decaying linear attention bias, but its heuristic design lacks guarantees for monotonic decay, leading to suboptimal performance on extremely long sequences.

— HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models

“However, the function rapidly approaches the zero point”

— Context-aware Biases for Length Extrapolation

“why it fails to retrieve information as it becomes local attention as the context length increases”

— Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation

“However, the function rapidly approaches the zero point~chi2022kerple.”

— MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation

Beaten on benchmarks

Head-to-head results where a newer method reports beating ALiBi. Values are copied from the source paper's tables — verify against the cited paper.

BiPE-RoPE beats ALiBi
22.36 vs 17.62
Average · [ALiBi baseline]
Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
DAPE-Kerple beats ALiBi
3.8642 vs 4.7679
perplexity (mean) · [training_length_512_eval_8192]
DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
CABLE beats ALiBi
21.36 vs 18.71
nDCG@10 · [BERT models trained on T=512]
Context-aware Biases for Length Extrapolation
HoPE beats ALiBi
19.34 vs 18.83
RGL · [GovReport task]
HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
MEP beats ALiBi
5.309 vs 5.440
Perplexity · [ArXiv, parameter-free]
MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.