DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs
This addresses the problem of efficient LLM deployment for practitioners, representing an incremental improvement in quantization techniques.
The paper tackles the deployment challenge of large language models (LLMs) due to hardware constraints by proposing density-aware post-training weight-only quantization (DAQ), which reduces perplexity loss by an average of 22.8% on LLaMA and 19.6% on LLaMA-2 compared to baseline methods.
Large language models (LLMs) excel in various tasks but face deployment challenges due to hardware constraints. We propose density-aware post-training weight-only quantization (DAQ), which has two stages: 1) density-centric alignment, which identifies the center of high-density weights and centers the dynamic range on this point to align high-density weight regions with floating-point high-precision regions; 2) learnable dynamic range adjustment, which adjusts the dynamic range by optimizing quantization parameters (i.e., scale and zero-point) based on the impact of weights on the model output. Experiments on LLaMA and LLaMA-2 show that DAQ consistently outperforms the best baseline method, reducing perplexity loss by an average of 22.8% on LLaMA and 19.6% on LLaMA-2. Our code is available at https://github.com/LuoYingSong/DAQ.