The Quarks of Attention
This work provides a theoretical foundation for attention mechanisms in deep learning, which is incremental as it builds on existing models to analyze computational properties.
The paper investigates the fundamental building blocks of attention in deep learning, classifying them by source, target, and mechanism, and identifies three key mechanisms: additive activation attention, multiplicative output attention, and multiplicative synaptic attention, with additive attention playing a central role in proofs of lower bounds and reducing circuit depth.
Attention plays a fundamental role in both natural and artificial intelligence systems. In deep learning, attention-based neural architectures, such as transformer architectures, are widely used to tackle problems in natural language processing and beyond. Here we investigate the fundamental building blocks of attention and their computational properties. Within the standard model of deep learning, we classify all possible fundamental building blocks of attention in terms of their source, target, and computational mechanism. We identify and study three most important mechanisms: additive activation attention, multiplicative output attention (output gating), and multiplicative synaptic attention (synaptic gating). The gating mechanisms correspond to multiplicative extensions of the standard model and are used across all current attention-based deep learning architectures. We study their functional properties and estimate the capacity of several attentional building blocks in the case of linear and polynomial threshold gates. Surprisingly, additive activation attention plays a central role in the proofs of the lower bounds. Attention mechanisms reduce the depth of certain basic circuits and leverage the power of quadratic activations without incurring their full cost.