PoTPTQ: A Two-step Power-of-Two Post-training for LLMs
This work addresses computational resource constraints for deploying LLMs, offering a novel method that enhances both accuracy and inference speed, though it appears incremental as it builds on existing PoT quantization techniques.
The paper tackles the challenge of deploying large language models (LLMs) by proposing a power-of-two (PoT) quantization framework that improves accuracy in low-precision formats (e.g., 2- and 3-bit) and achieves speedups of 3.67× on NVIDIA V100 and 1.63× on NVIDIA RTX 4090 compared to uniform integer dequantization.
Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing (NLP) tasks. However, their deployment is challenging due to the substantial computational resources required. Power-of-two (PoT) quantization is a general tool to counteract this difficulty. Albeit previous works on PoT quantization can be efficiently dequantized on CPUs using fixed-point addition, it showed less effectiveness on GPUs. The reason is entanglement of the sign bit and sequential bit manipulations needed for dequantization. We propose a novel POT quantization framework for LLM weights that (i) outperforms state-of-the-art accuracy in extremely low-precision number formats, and (ii) enables faster inference through more efficient dequantization. To maintain the accuracy of the quantized model, we introduce a two-step post-training algorithm: (i) initialize the quantization scales with a robust starting point, and (ii) refine these scales using a minimal calibration set. The performance of our PoT post-training algorithm surpasses the current state-of-the-art in integer quantization, particularly at low precisions such as 2- and 3-bit formats. Our PoT quantization accelerates the dequantization step required for the floating point inference and leads to $3.67\times$ speed up on a NVIDIA V100, and $1.63\times$ on a NVIDIA RTX 4090, compared to uniform integer dequantization.