Hybrid and Non-Uniform quantization methods using retro synthesis data for efficient inference
This work addresses the problem of efficient model inference for privacy-constrained applications by eliminating the need for training data in quantization, offering a strong specific gain for practitioners.
This paper proposes a data-independent post-training quantization scheme using 'Retro-Synthesis Data' generated from FP32 model layer statistics. This method outperformed state-of-the-art techniques like ZeroQ and DFQ for 8, 6, and 4-bit precisions on ImageNet and CIFAR-10 datasets, and also introduced Hybrid and Non-Uniform Quantization variants.
Existing quantization aware training methods attempt to compensate for the quantization loss by leveraging on training data, like most of the post-training quantization methods, and are also time consuming. Both these methods are not effective for privacy constraint applications as they are tightly coupled with training data. In contrast, this paper proposes a data-independent post-training quantization scheme that eliminates the need for training data. This is achieved by generating a faux dataset, hereafter referred to as Retro-Synthesis Data, from the FP32 model layer statistics and further using it for quantization. This approach outperformed state-of-the-art methods including, but not limited to, ZeroQ and DFQ on models with and without Batch-Normalization layers for 8, 6, and 4 bit precisions on ImageNet and CIFAR-10 datasets. We also introduced two futuristic variants of post-training quantization methods namely Hybrid Quantization and Non-Uniform Quantization