Transformer Based Linear Attention with Optimized GPU Kernel Implementation
This addresses the computational bottleneck for large-scale Transformer training and inference, offering significant practical gains for AI researchers and engineers.
The paper tackles the practical inefficiency of linear attention mechanisms in Transformers by proposing a novel method with optimized GPU implementation, achieving 3.3× speed improvement and 3.6× memory reduction while maintaining comparable accuracy to regular attention in a 1.4B parameter language model.
The original softmax-based attention mechanism (regular attention) in the extremely successful Transformer architecture computes attention between $N$ tokens, each embedded in a $D$-dimensional head, with a time complexity of $O(N^2D)$. Given the success of Transformers, improving their runtime during both training and inference is a popular research area. One such approach is the introduction of the linear attention (LA) mechanisms, which offers a linear time complexity of $O(ND^2)$ and have demonstrated comparable accuracy to regular attention. However, LA in practice lags behind its theoretical efficiency. We propose a novel method for LA's forward and backward passes, along with a highly-optimized CUDA implementation. Our approach outperforms the state-of-the-art by 3.3 times in speed and reduces memory consumption by 3.6 times. We validate these improvements in both single-layer and end-to-end settings by training a 1.4 billion parameter language model, which demonstrates similar expressivity to regular attention on major reasoning benchmarks.