Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models
This work addresses the need for efficient model compression in AI by improving pruning performance for large language models, though it is incremental as it builds on existing pruning techniques.
The paper tackled the problem of pruning large language models by proposing a non-uniform method based on Shapley values to assign tailored pruning budgets per layer, resulting in reductions in perplexity of 18.01% and 19.55% on LLaMA-7B and LLaMA-13B compared to SparseGPT at 70% sparsity.
Pruning large language models (LLMs) is a promising solution for reducing model sizes and computational complexity while preserving performance. Traditional layer-wise pruning methods often adopt a uniform sparsity approach across all layers, which leads to suboptimal performance due to the varying significance of individual transformer layers within the model not being accounted for. To this end, we propose the Shapley Value-based Non-Uniform Pruning (SV-NUP) method for LLMs. This approach quantifies the contribution of each transformer layer to the overall model performance, enabling the assignment of tailored pruning budgets to different layers to retain critical parameters. To further improve efficiency, we design the Sliding Window-based Shapley Value approximation method. It substantially reduces computational overhead compared to exact SV calculation methods. Extensive experiments on various LLMs including LLaMA-v1, LLaMA-v2 and OPT demonstrate the effectiveness of the proposed approach. The results reveal that non-uniform pruning significantly enhances the performance of pruned models. Notably, SV-NUP achieves a reduction in perplexity (PPL) of 18.01% and 19.55% on LLaMA-7B and LLaMA-13B, respectively, compared to SparseGPT at 70% sparsity.