Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
This work addresses the deployment of large ViTs on resource-constrained devices, offering a fully quantized solution with significant performance gains, though it is incremental as it builds on existing quantization methods.
The paper tackles the performance drop in low-bit quantized vision transformers (ViTs) by identifying and addressing information distortion in self-attention maps, achieving a 6.14x theoretical speedup and 80.9% Top-1 accuracy on ImageNet, surpassing full-precision counterparts by 1.0%.
The large pre-trained vision transformers (ViTs) have demonstrated remarkable performance on various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. Among the powerful compression approaches, quantization extremely reduces the computation and memory consumption by low-bit parameters and bit-wise operations. However, low-bit ViTs remain largely unexplored and usually suffer from a significant performance drop compared with the real-valued counterparts. In this work, through extensive empirical analysis, we first identify the bottleneck for severe performance drop comes from the information distortion of the low-bit quantized self-attention map. We then develop an information rectification module (IRM) and a distribution guided distillation (DGD) scheme for fully quantized vision transformers (Q-ViT) to effectively eliminate such distortion, leading to a fully quantized ViTs. We evaluate our methods on popular DeiT and Swin backbones. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, our Q-ViT can theoretically accelerates the ViT-S by 6.14x and achieves about 80.9% Top-1 accuracy, even surpassing the full-precision counterpart by 1.0% on ImageNet dataset. Our codes and models are attached on https://github.com/YanjingLi0202/Q-ViT