HALO: Learning to Prune Neural Networks with Shrinkage
This work addresses the need for efficient, high-accuracy neural networks in resource-constrained applications, representing an incremental improvement over existing sparsity methods.
The paper tackles the problem of reducing neural network size while maintaining accuracy by introducing the Hierarchical Adaptive Lasso (HALO) penalty, which learns to adaptively sparsify weights, resulting in highly sparse networks (e.g., 5% of parameters) with significant performance gains over state-of-the-art pruning methods at the same sparsity level.
Deep neural networks achieve state-of-the-art performance in a variety of tasks by extracting a rich set of features from unstructured data, however this performance is closely tied to model size. Modern techniques for inducing sparsity and reducing model size are (1) network pruning, (2) training with a sparsity inducing penalty, and (3) training a binary mask jointly with the weights of the network. We study different sparsity inducing penalties from the perspective of Bayesian hierarchical models and present a novel penalty called Hierarchical Adaptive Lasso (HALO) which learns to adaptively sparsify weights of a given network via trainable parameters. When used to train over-parametrized networks, our penalty yields small subnetworks with high accuracy without fine-tuning. Empirically, on image recognition tasks, we find that HALO is able to learn highly sparse network (only 5% of the parameters) with significant gains in performance over state-of-the-art magnitude pruning methods at the same level of sparsity. Code is available at https://github.com/skyler120/sparsity-halo.