UniPruning: Unifying Local Metric and Global Feedback for Scalable Sparse LLMs
This work addresses the problem of efficiently sparsifying LLMs for researchers and practitioners, offering a scalable solution that balances speed and robustness, though it is incremental in improving existing pruning methods.
The paper tackles the challenge of pruning large language models (LLMs) to reduce computational and memory costs by introducing UniPruning, a unified post-training framework that combines local saliency metrics with global coordination, achieving competitive or superior perplexity and zero-shot accuracy on multiple LLM families and benchmarks without updating model weights.
Large Language Models (LLMs) achieve strong performance across diverse tasks but face prohibitive computational and memory costs. Pruning offers a promising path by inducing sparsity while preserving architectural flexibility. However, existing methods struggle to balance efficiency and robustness: local metric approaches prune layer by layer but often collapse under high sparsity, whereas global feedback methods enforce consistency at the cost of expensive weight updates or restrictive semi-structured formats. We present UniPruning, a unified post-training pruning framework that combines the speed of local saliency metrics with the stability of global coordination, enabled by a mirror descent based optimization, all without updating model weights. UniPruning leverages fast layer-wise scoring and a lightweight global controller to allocate a single sparsity budget, supporting both unstructured and semi-structured N :M pruning within one framework. After a brief calibration, it can generate pruning masks for arbitrary sparsity levels in one shot, and adapts seamlessly to hardware-aware constraints. Extensive experiments on multiple pretrained LLM families and standard benchmarks show that UniPruning consistently delivers competitive or superior perplexity and zero-shot accuracy. Ablation studies further highlight the importance of mirror descent and local saliency anchoring. Overall, UniPruning provides an efficient, principled, and scalable solution for sparsifying large-scale LLMs. Our code is available at: https://github.com/RainbowQTT/UniPruning.