ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
This work addresses the efficiency and cost issues in deploying LLMs, offering a novel optimization approach for semi-structured pruning, though it is incremental as it builds on existing pruning methods.
The paper tackled the problem of inefficient and costly serving of large language models (LLMs) by proposing ProxSparse, a learning-based framework for semi-structured pruning mask selection, which consistently outperformed previous methods in evaluations on 7 models.
Large Language Models (LLMs) have demonstrated exceptional performance in natural language processing tasks, yet their massive size makes serving them inefficient and costly. Semi-structured pruning has emerged as an effective method for model acceleration, but existing approaches are suboptimal because they focus on local, layer-wise optimizations using heuristic rules, failing to leverage global feedback. We present ProxSparse, a learning-based framework for mask selection enabled by regularized optimization. ProxSparse transforms the rigid, non-differentiable mask selection process into a smoother optimization procedure, allowing gradual mask exploration with flexibility. ProxSparse does not involve additional weight updates once the mask is determined. Our extensive evaluations on 7 widely used models show that ProxSparse consistently outperforms previously proposed semi-structured mask selection methods with significant improvement, demonstrating the effectiveness of our learned approach towards semi-structured pruning.