Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation
This addresses the problem of high inference costs for deploying foundation models, offering a practical compression solution that is incremental in improving efficiency.
The paper tackles the challenge of compressing large language models for resource-constrained environments by introducing ACIP, a method that enables compression to any target size without re-computation, achieving state-of-the-art results on various LLMs and tasks.
The adoption of Foundation Models in resource-constrained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitations is post-training compression, which aims to balance reduced model size against performance degradation. This work presents Any Compression via Iterative Pruning (ACIP), a novel algorithmic approach to determine a compression-performance trade-off from a single stochastic gradient descent run. To achieve parameter efficiency, we use an SVD-reparametrization of linear layers and iteratively prune their singular values with a sparsity-inducing penalty. Importantly, the pruning order of the parameters is used to derive a global score map that allows compressing a model to any target size without re-computation. We evaluate ACIP on a large selection of open-weight LLMs and downstream tasks, demonstrating state-of-the-art results compared to existing factorization-based compression methods. We also show that ACIP seamlessly complements common quantization-based compression techniques.