Long-context / context-window extension
LongLoRA
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
Superseded baseline#13 of 53 most-superseded · first seen Sep 21, 2023
Superseded — cited as a baseline and beaten by newer methods
2 papers critique it · 3 beat it on benchmarks
What papers say
Verbatim critique sentences, each from a paper that cites LongLoRA as a baseline.
However, this sparse attention mechanism is not applicable during inference, necessitating a return to the original full attention post-training.
“these methods typically require finetuning to achieve extension, which can be resource and time-intensive given the quadratic complexity of Transformers”
Beaten on benchmarks
Head-to-head results where a newer method reports beating LongLoRA. Values are copied from the source paper's tables — verify against the cited paper.
FreqKV beats LongLoRA
7.47 vs 8.29
Perplexity · [7B, 32K training]
FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window ExtensionET beats LongLoRA
76.03 vs 72.95
Avg. · [RULER benchmark]
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
What to use instead
Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.
- May 14, 2026