LQER: Low-Rank Quantization Error Reconstruction for LLMs
This addresses the problem of efficient deployment of LLMs for practitioners by reducing hardware resource usage while maintaining performance, though it appears incremental as it builds on existing quantization techniques.
The paper tackles the challenge of post-training quantization for Large Language Models by introducing LQER, a method that combines quantization and low-rank approximation to achieve nearly-lossless W4A8 quantization on various LLMs and downstream tasks, using 1.36× fewer hardware resources than the leading state-of-the-art method.
Post-training quantization of Large Language Models (LLMs) is challenging. In this work, we introduce Low-rank Quantization Error Reduction (LQER), which combines quantization and low-rank approximation to recover the model capability. LQER leverages an activation-induced scale matrix to drive the singular value distribution of quantization error towards a desirable distribution, which enables nearly-lossless W4A8 quantization on various LLMs and downstream tasks without the need for knowledge distillation, grid search, or gradient-base iterative optimization. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes to collect high-precision weights from irregular memory locations. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using 1.36$\times$ fewer hardware resources than the leading state-of-the-art method. We open-source our framework at https://github.com/ChengZhang-98/lqer