Beyond One-Size-Fits-All Pruning via Evolutionary Metric Search for Large Language Models
This work addresses the need for efficient, adaptive pruning in LLMs to reduce computational costs, offering a cost-effective compression solution with cross-task and cross-model generalizability, though it is incremental as it builds on existing pruning and evolutionary optimization techniques.
The paper tackles the problem of fixed pruning strategies being inadequate for large language models (LLMs) due to weight distribution variations, and introduces OptiShear, an evolutionary optimization framework that adaptively prunes LLMs, demonstrating consistent outperformance over existing methods on models like LLaMA-1/2/3 and Mistral (7B-70B) across benchmarks.
Post-training pruning has emerged as a crucial optimization technique as large language models (LLMs) continue to grow rapidly. However, the significant variations in weight distributions across different LLMs make fixed pruning strategies inadequate for multiple models. In this paper, we introduce \textbf{\textsc{OptiShear}}, an efficient evolutionary optimization framework for adaptive LLM pruning. Our framework features two key innovations: an effective search space built on our Meta pruning metric to handle diverse weight distributions, and a model-wise reconstruction error for rapid evaluation during search trials. We employ Non-dominated Sorting Genetic Algorithm III (NSGA-III) to optimize both pruning metrics and layerwise sparsity ratios. Through extensive evaluation on LLaMA-1/2/3 and Mistral models (7B-70B) across multiple benchmarks, we demonstrate that our adaptive pruning metrics consistently outperform existing methods. Additionally, our discovered layerwise sparsity ratios enhance the effectiveness of other pruning metrics. The framework exhibits strong cross-task and cross-model generalizability, providing a cost-effective solution for model compression.