Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework
This work addresses the challenge of reducing inference costs in vision networks without retraining, offering a method that can improve upon existing pruning techniques by better preserving deep representations.
The authors tackled the problem of one-shot post-training pruning for vision networks by optimizing a global reconstruction objective that accounts for nonlinear activations deep in the network, achieving state-of-the-art results on benchmarks including residual networks and Vision Transformers with up to 304 million parameters.
We present SNOWS, a one-shot post-training pruning framework aimed at reducing the cost of vision network inference without retraining. Current leading one-shot pruning methods minimize layer-wise least squares reconstruction error which does not take into account deeper network representations. We propose to optimize a more global reconstruction objective. This objective accounts for nonlinear activations deep in the network to obtain a better proxy for the network loss. This nonlinear objective leads to a more challenging optimization problem -- we demonstrate it can be solved efficiently using a specialized second-order optimization framework. A key innovation of our framework is the use of Hessian-free optimization to compute exact Newton descent steps without needing to compute or store the full Hessian matrix. A distinct advantage of SNOWS is that it can be readily applied on top of any sparse mask derived from prior methods, readjusting their weights to exploit nonlinearities in deep feature representations. SNOWS obtains state-of-the-art results on various one-shot pruning benchmarks including residual networks and Vision Transformers (ViT/B-16 and ViT/L-16, 86m and 304m parameters respectively).