LoaQ: Layer-wise Output Approximation Quantization
This work addresses the challenge of improving quantization accuracy for LLMs, which is crucial for efficient deployment, though it is incremental as it builds on existing layer-wise PTQ frameworks.
The paper tackles the problem of insufficient output approximation in layer-wise post-training quantization for large language models by proposing LoaQ, which explicitly targets output-level consistency, resulting in enhanced quantization quality for models like LLaMA and Qwen in both weight-only and weight-activation joint quantization.
A natural and intuitive idea in model quantization is to approximate each component's quantized output to match its original. Layer-wise post-training quantization (PTQ), though based on this idea, adopts a strictly local view and can achieve, at best, only activation-aware approximations of weights. As a result, it often leads to insufficient approximations and practical deviations from this guiding intuition. Recent work has achieved a more accurate approximation of linear-layer outputs within the framework of layer-wise PTQ, but such refinements remain inadequate for achieving alignment with the full model output. Based on a deeper understanding of the structural characteristics of mainstream LLMs, we propose $LoaQ$, an output-approximation method for layer-wise PTQ that explicitly targets output-level consistency. It better aligns with this intuition and can feature a simple closed-form solution, making it orthogonal to existing techniques and readily integrable into existing quantization pipelines. Experiments on the LLaMA and Qwen model families demonstrate that LoaQ performs effectively in both weight-only and weight-activation joint quantization. By integrating seamlessly with existing quantization strategies, it further enhances overall quantization quality and shows strong potential to advance the frontier of post-training quantization.