GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
This work addresses the challenge of efficiently compressing large language models for deployment, offering incremental improvements over existing quantization techniques.
The paper tackles the problem of post-training quantization for large language models by proposing GuidedQuant, which integrates end loss gradient information and preserves weight dependencies, resulting in consistent performance boosts across various quantization types and outperforming existing non-uniform scalar quantization methods.
Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant.