Learning Best Combination for Efficient N:M Sparsity
This work addresses the computational inefficiency in training sparse models for deployment on hardware like NVIDIA A100 GPUs, offering a more efficient method compared to prior approaches that require expensive pre-training or dense gradients.
The paper tackles the problem of efficiently training N:M sparse neural networks, which restrict at most N non-zero weights in each block of M, by formulating it as a combinatorial optimization problem and solving it with a divide-and-conquer approach that learns scores for combination subsets, achieving consistent performance improvements over existing methods across various networks.
By forcing at most N out of M consecutive weights to be non-zero, the recent N:M network sparsity has received increasing attention for its two attractive advantages: 1) Promising performance at a high sparsity. 2) Significant speedups on NVIDIA A100 GPUs. Recent studies require an expensive pre-training phase or a heavy dense-gradient computation. In this paper, we show that the N:M learning can be naturally characterized as a combinatorial problem which searches for the best combination candidate within a finite collection. Motivated by this characteristic, we solve N:M sparsity in an efficient divide-and-conquer manner. First, we divide the weight vector into $C_{\text{M}}^{\text{N}}$ combination subsets of a fixed size N. Then, we conquer the combinatorial problem by assigning each combination a learnable score that is jointly optimized with its associate weights. We prove that the introduced scoring mechanism can well model the relative importance between combination subsets. And by gradually removing low-scored subsets, N:M fine-grained sparsity can be efficiently optimized during the normal training phase. Comprehensive experiments demonstrate that our learning best combination (LBC) performs consistently better than off-the-shelf N:M sparsity methods across various networks. Our project is released at \url{https://github.com/zyxxmu/LBC}.