POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models
This addresses the problem of high inference costs for users of large foundation models by enabling more efficient deployment without retraining, though it is an incremental improvement over existing pruning methods.
The paper tackles the inefficiency of static pruning in large foundation models by introducing POP, an online structural pruning framework that dynamically adjusts sparsity during autoregressive token generation, achieving higher accuracy with lower computational overhead and reduced inference latency across various model types.
Large foundation models (LFMs) achieve strong performance through scaling, yet current structural pruning methods derive fixed pruning decisions during inference, overlooking sparsity patterns that emerge in the autoregressive token generation. In this paper, we propose POP (Partition-guided Online Pruning), an efficient online structural pruning framework that enables context-conditioned dynamic pruning with minimal computational overhead. POP partitions model channels into retained, candidate, and pruned regions, where prefilling defines a coarse pruning partition, and the decoding stage generates a fine-grained mask within the candidate region, avoiding full-channel re-evaluation. The coarse pruning partition preserves consistently important weights, while the fine-grained masking provides context-conditioned variation during decoding. Moreover, POP is a lightweight, plug-and-play method that requires no preprocessing, including offline calibration, retraining, or learning predictors. Extensive evaluations across diverse LFMs, including large language models (LLMs), mixture-of-experts models (MoEs), and vision-language models (VLMs), demonstrate that POP consistently delivers higher accuracy than existing pruning approaches while incurring smaller computational overhead and minimizing inference latency.