Sliding Window Recurrences for Sequence Models
This addresses computational bottlenecks in large-scale language modeling, though it appears incremental as an optimization of existing recurrence methods.
The paper tackles the problem of improving efficiency in multi-hybrid sequence models by introducing Sliding Window Recurrences (SWR) that align with GPU memory hierarchies, achieving 10-40% speedup over optimized Transformers while maintaining perplexity.
Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.