Vision Big Bird: Random Sparsification for Full Attention
This addresses efficiency challenges in vision tasks for researchers and practitioners, but it is incremental as it adapts an existing NLP method to vision.
The paper tackles the high computational cost of global self-attention in Vision Transformers for high-resolution vision tasks by proposing a novel sparse attention mechanism based on Big Bird, which separates heads into groups for local features, random sparse attention, and global attention, achieving competitive performance and enabling removal of positional encoding.
Recently, Transformers have shown promising performance in various vision tasks. However, the high costs of global self-attention remain challenging for Transformers, especially for high-resolution vision tasks. Inspired by one of the most successful transformers-based models for NLP: Big Bird, we propose a novel sparse attention mechanism for Vision Transformers (ViT). Specifically, we separate the heads into three groups, the first group used convolutional neural network (CNN) to extract local features and provide positional information for the model, the second group used Random Sampling Windows (RS-Win) for sparse self-attention calculation, and the third group reduces the resolution of the keys and values by average pooling for global attention. Based on these components, ViT maintains the sparsity of self-attention while maintaining the merits of Big Bird (i.e., the model is a universal approximator of sequence functions and is Turing complete). Moreover, our results show that the positional encoding, a crucial component in ViTs, can be safely removed in our model. Experiments show that Vision Big Bird demonstrates competitive performance on common vision tasks.