Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models
This work addresses efficiency challenges in LLM inference for AI practitioners, offering a novel activation sparsity approach that is incremental but impactful for specific bottlenecks.
The paper tackled the problem of accelerating the prefill stage in large language models by introducing Amber Pruner, a training-free N:M activation sparsity method, which achieved effective sparsification and acceleration of over 55% of linear computations without retraining. It also integrated this with quantization to preserve performance across downstream tasks, showing advantages in generative tasks.
In the era of large language models (LLMs), N:M sparsity has emerged as a structured compression technique critical for accelerating inference. While prior work has primarily focused on weight sparsity, it often suffers from significant accuracy degradation. Activation sparsity, though promising, is typically training-dependent and faces challenges in generalization. To address these limitations, we introduce Amber Pruner, a training-free N:M activation sparsity method designed specifically for the prefill stage, targeting the acceleration of linear projection layers in LLMs. Extensive experiments across multiple models and sparsity ratios (2:4, 4:8, and 8:16) demonstrate that Amber Pruner can effectively sparsify and accelerate more than 55% of linear computations without requiring model retraining. To further enhance generality and efficiency, we propose Outstanding-sparse, a unified framework that integrates Amber Pruner with post-training W8A8 quantization. Our approach preserves strong performance across a range of downstream tasks, with notable advantages in generative tasks. This work pioneers a new frontier in activation sparsity, providing foundational insights that are poised to guide the co-evolution of algorithms and architectures in the design of next-generation AI systems.