Scholar Feed

Long-context / context-window extension

YaRN

YaRN: Efficient Context Window Extension of Large Language Models

Heavily superseded#2 of 53 most-superseded · first seen Aug 31, 2023

Heavily superseded — a standard baseline that newer methods routinely beat

10 papers critique it · 8 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites YaRN as a baseline.

traditional approaches chen2023extending often suffer from a significant performance drop chen2023clex, ding2024longrope at the target length due to their limited generalization capability.
— DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
“the efficiency is relatively low. For example, to reach the context length of 128K tokens, using YaRN, one has to pretrain an LLM on 64K tokens.”
— Stacked from One: Multi-Scale Self-Injection for Context Window Extension
“rescaling factors derived from previous methods often fall short of achieving the effective target context length.”
— LongRoPE2: Near-Lossless LLM Context Window Scaling
“PI and YaRN suffer from slow motion, leading to lower Dynamic Degree”
— RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers
“However, these static approaches do not account for the distinctive spectral progression of the diffusion process, where low-frequency structures are generated in the first sampling steps, while high-frequency details are resolved later”
— DyPE: Dynamic Position Extrapolation for Ultra High Resolution Diffusion
“these methods typically require finetuning to achieve extension, which can be resource and time-intensive given the quadratic complexity of Transformers”
— LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
“methods like NTK, Dyn-NTK, and YaRN suffer from attention logit outliers due to their positional embedding interpolations”
— A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)
“While NTK-By-Parts and YaRN have lower perplexity in language modeling tasks, PI has better fine-tuning performance on long-context downstream tasks that are more related to practical scenarios.”
— Extending LLMs' Context Window with 100 Samples
“although YaRN improves the length extrapolation capability of RoPE to some extent, it still suffers from performance drop when being evaluated on very long sequences”
— Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
“these methods typically require additional fine-tuning on longer texts and have explicit extrapolation upper bounds.”
— LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training

Beaten on benchmarks

Head-to-head results where a newer method reports beating YaRN. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.