CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
This addresses the challenge of deploying quantized models in real-world scenarios where calibration data is inaccessible, offering a calibration-free alternative that matches methods requiring such data.
The paper tackles the problem of post-training quantization for large language models without needing calibration data, which is often unavailable or private, by proposing algorithms for learned transformations and adaptive rounding, resulting in improved benchmark scores for Gemma 2 models, such as increasing from 61.9 to 62.4 for 4-bit and from 52.0 to 60.6 for 3-bit quantization with minimal computational overhead.
Post-training quantization is an effective method for reducing the serving cost of large language models, where the standard approach is to use a round-to-nearest quantization level scheme. However, this often introduces large errors due to outliers in the weights. Proposed mitigation mechanisms include applying adaptive rounding, random rotation transformations or committing to a post-training target using calibration data. Unfortunately, this reliance on calibration data can be severely limiting in some real-world scenarios as such data may be unavailable or subject to privacy regulations. In this paper, we propose algorithms to optimize transformations and adaptive rounding without access to any calibration data. The optimization is achieved by designing a suitable proxy function for the quantization loss without calibration data. To maintain inference efficiency, we perform structured matrix transformations for single matrices. For paired weights that interact directly in the computation graph, we use dual matrix transformations and adaptive rounding methods. We conduct experiments on Gemma 2 models, and observe consistent improvement over the baselines. For Gemma 2 9B quantization, our method improves the average benchmark score from 61.9 to 62.4 for 4-bit quantization and from 52.0 to 60.6 for 3-bit quantization, while adding less than 3% of computation overhead. Furthermore, our method achieves performance comparable to the commonly used GPTQ method, which requires calibration data.