Linear Log-Normal Attention with Unbiased Concentration
This addresses scalability issues for long sequences in transformers, though it is incremental as it builds on existing linearized attention methods.
The paper tackles the quadratic complexity of self-attention in transformers by proposing Linear Log-Normal Attention, which emulates the distribution and concentration of original attention, and shows it outperforms other linearized alternatives on natural language benchmarks.
Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models.