SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
This addresses the challenge of efficient LLM deployment for users needing reduced computational costs without expensive retraining, though it is incremental as it builds on existing one-shot methods.
The paper tackles the problem of high memory consumption and slow inference in large language models (LLMs) by introducing SLiM, a one-shot compression framework that integrates quantization, sparsity, and low-rank approximation, achieving up to 5.66% accuracy improvement for LLaMA-2-7B with 2:4 sparsity and 4-bit quantization, along with up to 4.3x speedup and 0.23x memory reduction.
Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to preserve accuracy. In contrast, one-shot compression methods eliminate retraining cost, but struggle to achieve accuracy comparable to dense models. This paper presents SLIM, a new one-shot compression framework that holistically integrates hardware-friendly quantization, sparsity, and low-rank approximation into a unified process. First, we formulate the quantization process using a probabilistic approach (SLIM-Quant) that enables us to apply uniform quantization. Then, we use an existing one-shot pruning method to apply semi-structured sparsity on top of the quantized weights. Finally, to compensate for the introduced aggregated quantization and sparsity error, we use a novel saliency function with unique invertible and additive features that enables us to mathematically compute the value of low-rank adapters. SLIM improves model accuracy by up to 5.66% (LLaMA-2-7B) for 2:4 sparsity with 4-bit weight quantization, outperforming prior methods. Models compressed with SLIM achieve up to 4.3x and 3.8x on Nvidia RTX3060 and A100 GPUs, respectively. Additionally, they achieve up to 0.23x end-to-end memory reduction in comparison to their dense counterparts. We also propose an optional PEFT recipe that further improves accuracy by up to 1.66% (LLaMA-2-13B) compared to SLIM without fine-tuning.