Scholar Feed

Long-context / context-window extension

RoPE

RoFormer: Enhanced Transformer with Rotary Position Embedding

Heavily superseded#1 of 53 most-superseded · first seen Apr 20, 2021

Heavily superseded — a standard baseline that newer methods routinely beat

17 papers critique it · 8 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites RoPE as a baseline.

RoPE-based language models have poor length generalization.
— DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
“However, RoPE can only operates on global angles, rendering relative angles implicit and inaccessible. Thus, RoPE struggles with periodic angular relations essential in trajectory prediction since it fails to address modular transformations”
— DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling
“Although effective, RoPE still relies on predefined static frequency patterns that are uniform across different inputs and attention heads. As a result, it remains position-dependent but not token- or context-dependent, limiting its expressiveness in modeling more nuanced sequence structures.”
— Context-aware Rotary Position Embedding
“the inherent flaw of rotary position embedding (RoPE) being used”
— Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
“However, RoPE's 1D design, effective for text, overlooks the spatiotemporal structure of video data, limiting its suitability for Video-LLMs.”
— VRoPE: Rotary Position Embedding for Video Large Language Models
“However, RoPE exhibits oscillatory attention patterns due to its trigonometric periodicity, which can destabilize long-distance dependency modeling barbero2024roundroundgomakes.”
— HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
“positional attention collapse, induced by the inherent locality bias of RoPE”
— Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models
“However, barbero2024round later provided a mathematical analysis showing that this claim is flawed: attention weights under RoPE do not necessarily decay proportionally with relative query-key distances.”
— Context-aware Biases for Length Extrapolation
“it does not decouple content and position semantically”
— Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
“RoPE suffers from long-term decay, as shown in Figure~fig:correlation(c), implying that as the relative distance increases, the relative upper bound on token correlations at modeled relative positions will continuously decrease.”
— 3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
“We hypothesize that, for long distance attention, the way that RoPE rotates the query and the key vectors may prevent the model from utilizing the dimensions that it rotates significantly.”
— The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
“The essential components (i.e., the RoPE matrices) of previous RoPE approaches rely on 2D rotation groups, which simplify computations but consequently restrict their feature projection capabilities, especially in high-dimensional spaces”
— ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices

Beaten on benchmarks

Head-to-head results where a newer method reports beating RoPE. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.