Instance-Aware Group Quantization for Vision Transformers
This addresses the challenge of efficiently compressing ViTs for deployment in resource-constrained environments, representing an incremental improvement over existing PTQ methods by adapting them to transformer-specific architectures.
The paper tackles the problem of severe performance degradation when applying post-training quantization (PTQ) methods from CNNs to vision transformers (ViTs) due to varying activation distributions per input instance, and introduces instance-aware group quantization (IGQ-ViT) that dynamically groups channels and adjusts group numbers under a bit-operation constraint, achieving effective results across image classification, object detection, and instance segmentation tasks with various transformer architectures.
Post-training quantization (PTQ) is an efficient model compression technique that quantizes a pretrained full-precision model using only a small calibration set of unlabeled samples without retraining. PTQ methods for convolutional neural networks (CNNs) provide quantization results comparable to full-precision counterparts. Directly applying them to vision transformers (ViTs), however, incurs severe performance degradation, mainly due to the differences in architectures between CNNs and ViTs. In particular, the distribution of activations for each channel vary drastically according to input instances, making PTQ methods for CNNs inappropriate for ViTs. To address this, we introduce instance-aware group quantization for ViTs (IGQ-ViT). To this end, we propose to split the channels of activation maps into multiple groups dynamically for each input instance, such that activations within each group share similar statistical properties. We also extend our scheme to quantize softmax attentions across tokens. In addition, the number of groups for each layer is adjusted to minimize the discrepancies between predictions from quantized and full-precision models, under a bit-operation (BOP) constraint. We show extensive experimental results on image classification, object detection, and instance segmentation, with various transformer architectures, demonstrating the effectiveness of our approach.