DnA: Denoising Attention for Visual Tasks
For researchers in visual perception tasks, DnA offers a simple modification to attention that yields consistent, though modest, improvements across image and video understanding.
The authors propose Denoising Attention (DnA) to replace softmax in multihead attention, reducing noisy attention patterns. DnA achieves a 0.8% absolute gain on ImageNet-1K with ViT-B, and improvements of 1.8% on video transformers and 0.5% on video LLMs.
The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.