Dynamic N:M Fine-grained Structured Sparse Attention Mechanism
This work addresses the problem of latency-sensitive applications for AI practitioners by providing a practical speedup method for Transformers, though it is incremental as it builds on existing sparse attention techniques.
The paper tackles the high complexity of attention mechanisms in Transformers by introducing DFSS, a dynamic N:M fine-grained structured sparse attention mechanism that achieves 1.27~1.89x speedups over full attention while maintaining accuracy comparable to full attention after a few finetuning epochs.
Transformers are becoming the mainstream solutions for various tasks like NLP and Computer vision. Despite their success, the high complexity of the attention mechanism hinders them from being applied to latency-sensitive tasks. Tremendous efforts have been made to alleviate this problem, and many of them successfully reduce the asymptotic complexity to linear. Nevertheless, most of them fail to achieve practical speedup over the original full attention under moderate sequence lengths and are unfriendly to finetuning. In this paper, we present DFSS, an attention mechanism that dynamically prunes the full attention weight matrix to N:M fine-grained structured sparse pattern. We provide both theoretical and empirical evidence that demonstrates DFSS is a good approximation of the full attention mechanism. We propose a dedicated CUDA kernel design that completely eliminates the dynamic pruning overhead and achieves speedups under arbitrary sequence length. We evaluate the 1:2 and 2:4 sparsity under different configurations and achieve 1.27~ 1.89x speedups over the full-attention mechanism. It only takes a couple of finetuning epochs from the pretrained model to achieve on par accuracy with full attention mechanism on tasks from various domains under different sequence lengths from 384 to 4096.