PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization
This work addresses the challenge of efficiently compressing vision transformers for deployment, offering a practical solution for computer vision applications, though it is incremental as it builds on existing quantization techniques.
The paper tackled the problem of post-training quantization for vision transformers, which previously suffered from significant accuracy drops, by proposing a twin uniform quantization method and a Hessian guided metric, achieving near-lossless accuracy with less than 0.5% drop at 8-bit quantization on ImageNet.
Quantization is one of the most effective methods to compress neural networks, which has achieved great success on convolutional neural networks (CNNs). Recently, vision transformers have demonstrated great potential in computer vision. However, previous post-training quantization methods performed not well on vision transformer, resulting in more than 1% accuracy drop even in 8-bit quantization. Therefore, we analyze the problems of quantization on vision transformers. We observe the distributions of activation values after softmax and GELU functions are quite different from the Gaussian distribution. We also observe that common quantization metrics, such as MSE and cosine distance, are inaccurate to determine the optimal scaling factor. In this paper, we propose the twin uniform quantization method to reduce the quantization error on these activation values. And we propose to use a Hessian guided metric to evaluate different scaling factors, which improves the accuracy of calibration at a small cost. To enable the fast quantization of vision transformers, we develop an efficient framework, PTQ4ViT. Experiments show the quantized vision transformers achieve near-lossless prediction accuracy (less than 0.5% drop at 8-bit quantization) on the ImageNet classification task.