pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training
This work provides an incremental improvement in the accuracy and scalability of extremely low-bit quantized LLMs, which is important for deploying these models on edge devices.
This paper addresses the challenge of building efficient large language models (LLMs) with extremely low-bit weights (sub 2-bit) for edge deployment, which existing methods struggle with in terms of accuracy and scalability. The authors propose pQuant, a method that decouples linear layers into a 1-bit branch for efficient computation and a high-precision branch for sensitive parameters, achieving state-of-the-art performance in extremely low-bit quantization.
Quantization-Aware Training from scratch has emerged as a promising approach for building efficient large language models (LLMs) with extremely low-bit weights (sub 2-bit), which can offer substantial advantages for edge deployment. However, existing methods still fail to achieve satisfactory accuracy and scalability. In this work, we identify a parameter democratization effect as a key bottleneck: the sensitivity of all parameters becomes homogenized, severely limiting expressivity. To address this, we propose pQuant, a method that decouples parameters by splitting linear layers into two specialized branches: a dominant 1-bit branch for efficient computation and a compact high-precision branch dedicated to preserving the most sensitive parameters. Through tailored feature scaling, we explicitly guide the model to allocate sensitive parameters to the high-precision branch. Furthermore, we extend this branch into multiple, sparsely-activated experts, enabling efficient capacity scaling. Extensive experiments indicate our pQuant achieves state-of-the-art performance in extremely low-bit quantization.