FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation
This work addresses the challenge of efficient model compression for Vision Transformers, which is incremental as it builds on existing block-wise reconstruction frameworks to enhance quantization performance.
The paper tackles the problem of significant accuracy degradation in post-training quantization for Vision Transformers, especially under low-bit settings, by proposing FIMA-Q, a method that uses Fisher Information Matrix approximation to improve quantization loss, resulting in substantial accuracy gains over state-of-the-art approaches.
Post-training quantization (PTQ) has stood out as a cost-effective and promising model compression paradigm in recent years, as it avoids computationally intensive model retraining. Nevertheless, current PTQ methods for Vision Transformers (ViTs) still suffer from significant accuracy degradation, especially under low-bit quantization. To address these shortcomings, we analyze the prevailing Hessian-guided quantization loss, and uncover certain limitations of conventional Hessian approximations. By following the block-wise reconstruction framework, we propose a novel PTQ method for ViTs, dubbed FIMA-Q. Specifically, we firstly establish the connection between KL divergence and FIM, which enables fast computation of the quantization loss during reconstruction. We further propose an efficient FIM approximation method, namely DPLR-FIM, by employing the diagonal plus low-rank principle, and formulate the ultimate quantization loss. Our extensive experiments, conducted across various vision tasks with representative ViT-based architectures on public datasets, demonstrate that our method substantially promotes the accuracy compared to the state-of-the-art approaches, especially in the case of low-bit quantization. The source code is available at https://github.com/ShiheWang/FIMA-Q.