Agglomerative Attention
This addresses a key bottleneck for scaling transformer models, enabling larger and more efficient neural networks, though it is incremental as it builds on existing attention mechanisms.
The paper tackles the quadratic scaling problem of full attention in transformers by proposing an attention model with linear memory and computational requirements, achieving comparable performance to full attention networks on language modeling tasks.
Neural networks using transformer-based architectures have recently demonstrated great power and flexibility in modeling sequences of many types. One of the core components of transformer networks is the attention layer, which allows contextual information to be exchanged among sequence elements. While many of the prevalent network structures thus far have utilized full attention -- which operates on all pairs of sequence elements -- the quadratic scaling of this attention mechanism significantly constrains the size of models that can be trained. In this work, we present an attention model that has only linear requirements in memory and computation time. We show that, despite the simpler attention model, networks using this attention mechanism can attain comparable performance to full attention networks on language modeling tasks.