Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs
This work addresses the problem of efficient LLM compression for deployment, offering a novel joint approach that is incremental but provides strong practical gains.
The paper tackles the challenge of jointly applying quantization and sparsity to compress Large Language Models, which have conflicting requirements, by proposing Optimal Brain Restoration (OBR), a training-free framework that minimizes performance degradation through error compensation, achieving up to 4.72x speedup and 6.4x memory reduction with W4A4KV4 quantization and 50% sparsity.
Recent advances in Large Language Model (LLM) compression, such as quantization and pruning, have achieved notable success. However, as these techniques gradually approach their respective limits, relying on a single method for further compression has become increasingly challenging. In this work, we explore an alternative solution by combining quantization and sparsity. This joint approach, though promising, introduces new difficulties due to the inherently conflicting requirements on weight distributions: quantization favors compact ranges, while pruning benefits from high variance. To attack this problem, we propose Optimal Brain Restoration (OBR), a general and training-free framework that aligns pruning and quantization by error compensation between both. OBR minimizes performance degradation on downstream tasks by building on a second-order Hessian objective, which is then reformulated into a tractable problem through surrogate approximation and ultimately reaches a closed-form solution via group error compensation. Experiments show that OBR enables aggressive W4A4KV4 quantization with 50% sparsity on existing LLMs, and delivers up to 4.72x speedup and 6.4x memory reduction compared to the FP16-dense baseline.