Long-context / context-window extension

RoFormer: Enhanced Transformer with Rotary Position Embedding

17 critique · 8 beaten on benchmarks

YaRN: Efficient Context Window Extension of Large Language Models

10 critique · 8 beaten on benchmarks

Extending Context Window of Large Language Models via Positional Interpolation

8 critique · 5 beaten on benchmarks

APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding

8 critique · 3 beaten on benchmarks

Efficient Streaming Language Models with Attention Sinks

3 critique · 7 beaten on benchmarks

4 critique · 6 beaten on benchmarks

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

4 critique · 5 beaten on benchmarks

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

5 critique · 4 beaten on benchmarks

H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

5 critique · 3 beaten on benchmarks

Long Context Compression with Activation Beacon

2 critique · 4 beaten on benchmarks

1 critique · 4 beaten on benchmarks

KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation

2 critique · 3 beaten on benchmarks