Linear Self-Attention Approximation via Trainable Feedforward Kernel
This work addresses efficiency issues for researchers and practitioners using large-scale Transformer models, but it appears incremental as it builds on existing kernelized approaches.
The paper tackles the problem of reducing the quadratic computational complexity of self-attention in Transformers by proposing a trainable kernel method to approximate it with linear complexity, aiming to retain high accuracy.
In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches -- models attaining sub-quadratic attention complexity can utilize a notion of sparsity or a low-rank approximation of inputs to reduce the number of attended keys; other ways to reduce complexity include locality-sensitive hashing, key pooling, additional memory to store information in compacted or hybridization with other architectures, such as CNN. Often based on a strong mathematical basis, kernelized approaches allow for the approximation of attention with linear complexity while retaining high accuracy. Therefore, in the present paper, we aim to expand the idea of trainable kernel methods to approximate the self-attention mechanism of the Transformer architecture.