CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search
For practitioners deploying LLMs, CRePE offers a more effective pruning method with reduced search overhead, though the improvement is incremental over the state-of-the-art RIA method.
CRePE improves post-training pruning of LLMs by incorporating 2D local neighborhood context and adaptive coefficients into relative importance scoring, achieving consistent accuracy gains over existing methods. The PHO method reduces hyperparameter search time from ~11 hours to ~20 minutes while maintaining strong generalization across models.
Deploying Large Language Models (LLMs) in practice incurs substantial memory and computational costs. Post-training pruning (PTP) is an effective approach to reducing these costs by removing weights without additional training. Among existing methods, RIA introduces relative importance scores normalized by row and column sums, achieving state-of-the-art accuracy. However, RIA considers only 1D cross-shaped (row/column) directional information and assigns equal weight to row and column contributions. In this paper, we propose \textbf{CRePE}, which incorporates 2D local neighborhood context and adaptive coefficients into Relative Importance scoring. CRePE consistently outperforms existing PTP methods across diverse models and sparsity settings. However, identifying optimal adaptive coefficients via perplexity (PPL)-based hill climbing requires numerous PPL evaluations and approximately 11 hours of search time. To address this, we propose \textbf{PHO} (Proxy-based Hyperparameter Optimization), which eliminates the need for repeated PPL measurements and reduces the search time to approximately 20 minutes. Furthermore, the optimal hyperparameter configuration found by PHO on one model transfers well to other models, demonstrating strong generalization. Finally, we verify that CRePE can be orthogonally combined with existing techniques including Channel Permutation, non-uniform sparsity allocation, and re-pruning methods.