Efficiently Training A Flat Neural Network Before It has been Quantizated
This work addresses the efficiency of compressing vision transformer models for deployment, but it is incremental as it builds on existing post-training quantization methods by focusing on error modeling.
The paper tackles the problem of high quantization error in post-training quantization for vision transformers by discovering that a flat full-precision neural network is crucial for low-bit quantization, and proposes a framework that models activation and weight quantization errors as Gaussian noises to achieve this, with experimental results showing effectiveness.
Post-training quantization (PTQ) for vision transformers (ViTs) has garnered significant attention due to its efficiency in compressing models. However, existing methods typically overlook the relationship between a well-trained NN and the quantized model, leading to considerable quantization error for PTQ. However, it is unclear how to efficiently train a model-agnostic neural network which is tailored for a predefined precision low-bit model. In this paper, we firstly discover that a flat full precision neural network is crucial for low-bit quantization. To achieve this, we propose a framework that proactively pre-conditions the model by measuring and disentangling the error sources. Specifically, both the Activation Quantization Error (AQE) and the Weight Quantization Error (WQE) are statistically modeled as independent Gaussian noises. We study several noise injection optimization methods to obtain a flat minimum. Experimental results attest to the effectiveness of our approach. These results open novel pathways for obtaining low-bit PTQ models.