FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
This work solves the problem of efficient deployment of vision transformers for real-world applications by enabling lossless quantization, which is incremental as it builds on existing quantization methods but specifically targets transformers.
The paper tackles the problem of severe performance degradation in fully quantized vision transformers by addressing inter-channel variation in LayerNorm inputs and non-uniform distribution in attention maps, achieving lossless accuracy degradation (~1%) and outperforming previous works with lower bit-width on attention maps, e.g., 84.89% top-1 accuracy with ViT-L on ImageNet.
Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.