Pointer: Linear-Complexity Long-Range Modeling without Pre-training
This addresses the efficiency bottleneck for long-range modeling in scenarios where pre-training is impractical, offering a domain-specific improvement.
The paper tackles the problem of high computational complexity in long-range sequence modeling by introducing Pointer, a novel architecture that achieves linear O(NK) complexity while maintaining >95% accuracy on copy tasks at distances up to 2048 tokens and 2-10× speedup compared to standard transformers.
We introduce Pointer, a novel architecture that achieves linear $O(NK)$ complexity for long-range sequence modeling while maintaining superior performance without requiring pre-training. Unlike standard attention mechanisms that compute $O(N^2)$ pairwise interactions, our approach uses layer-wise pointer chaining where each layer's pointer selection depends on previous layer's pointer positions, creating explicit long-distance connections through pointer chains. We demonstrate that this architecture achieves $2$--$10\times$ speedup on long sequences compared to standard transformers, maintains $>95\%$ accuracy on copy tasks at distances up to 2048 tokens, and learns interpretable pointer patterns that reveal structured dependency modeling. Our experiments on efficiency benchmarks, long-range dependency tasks, and interpretability analysis show that Pointer offers a compelling alternative to attention mechanisms for scenarios requiring efficient long-range modeling without pre-training dependencies.