Long-context / context-window extension
Activation Beacon
Long Context Compression with Activation Beacon
Superseded — cited as a baseline and beaten by newer methods
2 papers critique it · 4 beat it on benchmarks
What papers say
Verbatim critique sentences, each from a paper that cites Activation Beacon as a baseline.
However, they need a copy of multi-head attention, which amounts to approximately 2B for 7B models.
“the compressed length still grows linearly with the original context length. This fails to fundamentally alter the asymptotic order of spatiotemporal complexity and can only improve efficiency by reducing constant factors”
Beaten on benchmarks
Head-to-head results where a newer method reports beating Activation Beacon. Values are copied from the source paper's tables — verify against the cited paper.
CoMeT beats Activation Beacon
40.10 vs 30.71
Avg · [SCROLLS benchmark, 3k memory budget]
CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling3D-RPE-LLaMA2-7B beats Activation Beacon
2.72 vs 3.45
Perplexity · [Proof-Pile at 8k context]
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position EncodingSharedLLM beats Activation Beacon
30.93 vs 28.44
Accuracy · [LLaMA-2 MDQA]
Stacked from One: Multi-Scale Self-Injection for Context Window Extension
What to use instead
Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.