Fast Monte-Carlo Approximation of the Attention Mechanism
This addresses efficiency problems for users of large Transformer models, offering a method that is incremental as it builds on existing attention optimization without requiring model changes.
The paper tackles the high computational cost of self-attention in Transformers by introducing Monte-Carlo Attention (MCA), a randomized approximation method that reduces attention complexity by up to 11× in FLOPS on GLUE benchmarks while maintaining model accuracy.
We introduce Monte-Carlo Attention (MCA), a randomized approximation method for reducing the computational cost of self-attention mechanisms in Transformer architectures. MCA exploits the fact that the importance of each token in an input sequence varies with respect to their attention scores; thus, some degree of error can be tolerable when encoding tokens with low attention. Using approximate matrix multiplication, MCA applies different error bounds to encode input tokens such that those with low attention scores are computed with relaxed precision, whereas errors of salient elements are minimized. MCA can operate in parallel with other attention optimization schemes and does not require model modification. We study the theoretical error bounds and demonstrate that MCA reduces attention complexity (in FLOPS) for various Transformer models by up to 11$\times$ in GLUE benchmarks without compromising model accuracy.