Single Shot Structured Pruning Before Training
This addresses the computational bottleneck in training and inference for deep learning practitioners, offering a fast and easy-to-implement solution.
The paper tackles the problem of accelerating deep neural networks by introducing a structured pruning method applied before training, which speeds up training by 2x and inference by 3x.
We introduce a method to speed up training by 2x and inference by 3x in deep neural networks using structured pruning applied before training. Unlike previous works on pruning before training which prune individual weights, our work develops a methodology to remove entire channels and hidden units with the explicit aim of speeding up training and inference. We introduce a compute-aware scoring mechanism which enables pruning in units of sensitivity per FLOP removed, allowing even greater speed ups. Our method is fast, easy to implement, and needs just one forward/backward pass on a single batch of data to complete pruning before training begins.