Bi-ViT: Pushing the Limit of Vision Transformer Quantization
This work addresses the problem of deploying large pre-trained vision transformers on resource-limited devices by pushing quantization to its limit, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the challenge of fully binarizing vision transformers (ViTs), which previously suffered from unacceptable performance due to attention distortion from gradient vanishing and ranking disorder, and achieves significant improvements, such as 22.1% and 21.4% higher Top-1 accuracy over baselines with DeiT-Tiny and Swin-Tiny on ImageNet, along with 61.5x and 56.1x theoretical acceleration in FLOPs.
Vision transformers (ViTs) quantization offers a promising prospect to facilitate deploying large pre-trained networks on resource-limited devices. Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its limit remain largely unexplored and a very challenging task yet, due to their unacceptable performance. Through extensive empirical analyses, we identify the severe drop in ViT binarization is caused by attention distortion in self-attention, which technically stems from the gradient vanishing and ranking disorder. To address these issues, we first introduce a learnable scaling factor to reactivate the vanished gradients and illustrate its effectiveness through theoretical and experimental analyses. We then propose a ranking-aware distillation method to rectify the disordered ranking in a teacher-student framework. Bi-ViT achieves significant improvements over popular DeiT and Swin backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny and Swin-Tiny, our method significantly outperforms baselines by 22.1% and 21.4% respectively, while 61.5x and 56.1x theoretical acceleration in terms of FLOPs compared with real-valued counterparts on ImageNet.