Training-Free Activation Sparsity in Large Language Models
This addresses the efficiency bottleneck for LLM inference by providing a training-free sparsity method, which is incremental as it builds on existing sparsity concepts but simplifies implementation.
The paper tackles the problem of enabling practical inference speedups in large language models (LLMs) through activation sparsity without training, and the result is TEAL, a method that achieves 40-50% model-wide sparsity with minimal performance degradation and demonstrates decoding speed-ups of up to 1.53x and 1.8x.
Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for matrix multiplications during the forward pass. However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. This paper describes TEAL, a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53$\times$ and 1.8$\times$ at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.