SPDY: Accurate Pruning with Speedup Guarantees
This addresses the efficiency gap in deep learning for practitioners by providing a pruning method that directly optimizes for inference time, though it is incremental as it builds on existing pruning approaches.
The paper tackles the problem of optimizing deep neural network pruning for inference speedup rather than just model size, introducing SPDY, a method that automatically determines layer-wise sparsity targets to achieve desired speedups while minimizing accuracy loss. Experiments show SPDY guarantees speedups and recovers higher accuracy than existing strategies across vision and language models, achieving the best known accuracy recovery for GPU-supported 2:4 sparsity with very little data.
The recent focus on the efficiency of deep neural networks (DNNs) has led to significant work on model compression approaches, of which weight pruning is one of the most popular. At the same time, there is rapidly-growing computational support for efficiently executing the unstructured-sparse models obtained via pruning. Yet, most existing pruning methods minimize just the number of remaining weights, i.e. the size of the model, rather than optimizing for inference time. We address this gap by introducing SPDY, a new compression method which automatically determines layer-wise sparsity targets achieving a desired inference speedup on a given system, while minimizing accuracy loss. SPDY is composed of two new techniques: the first is an efficient dynamic programming algorithm for solving the speedup-constrained layer-wise compression problem assuming a set of given layer-wise sensitivity scores; the second is a local search procedure for determining accurate layer-wise sensitivity scores. Experiments across popular vision and language models show that SPDY guarantees speedups while recovering higher accuracy relative to existing strategies, both for one-shot and gradual pruning scenarios, and is compatible with most existing pruning approaches. We also extend our approach to the recently-proposed task of pruning with very little data, where we achieve the best known accuracy recovery when pruning to the GPU-supported 2:4 sparsity pattern.