Static Key Attention in Vision
This incremental improvement offers a more efficient attention mechanism for vision models, potentially reducing computational overhead.
The paper tackles the problem of simplifying the attention mechanism in Vision Transformers by replacing dynamic keys with static ones, finding that this static key attention matches or exceeds standard self-attention performance in various vision tasks.
The success of vision transformers is widely attributed to the expressive power of their dynamically parameterized multi-head self-attention mechanism. We examine the impact of substituting the dynamic parameterized key with a static key within the standard attention mechanism in Vision Transformers. Our findings reveal that static key attention mechanisms can match or even exceed the performance of standard self-attention. Integrating static key attention modules into a Metaformer backbone, we find that it serves as a better intermediate stage in hierarchical hybrid architectures, balancing the strengths of depth-wise convolution and self-attention. Experiments on several vision tasks underscore the effectiveness of the static key mechanism, indicating that the typical two-step dynamic parameterization in attention can be streamlined to a single step without impacting performance under certain circumstances.