SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models
This addresses efficiency problems for AI practitioners deploying LLMs, offering a novel compression method with strong performance gains.
The paper tackles the deployment challenges of large language models by proposing SLaB, a framework that decomposes linear layer weights into sparse, low-rank, and binary components without retraining, achieving up to 36% lower perplexity and 8.98% higher accuracy on zero-shot tasks at 50% compression.
The rapid growth of large language models (LLMs) presents significant deployment challenges due to their massive computational and memory demands. While model compression, such as network pruning, offers potential solutions, most existing methods often fail to maintain good performance at high compression ratios. To address this, we propose SLaB, a novel framework that decomposes each linear layer weight into three complementary components: a sparse matrix, a low-rank matrix, and a binary matrix. SLaB eliminates the need for retraining and leverages activation-aware pruning scores to guide the decomposition process. Experiments on Llama-family models demonstrate that SLaB achieves state-of-the-art performance, reducing perplexity by up to 36% compared to existing methods at 50% compression and improving accuracy by up to 8.98% over the baseline on zero-shot tasks.