Differentiable Mask for Pruning Convolutional and Recurrent Networks
This provides a general pruning method for both vision and text architectures, addressing the need for model compression in multi-modal multi-task learning.
The paper tackles the problem of pruning deep networks for edge devices by introducing a differentiable mask that induces sparsity across various granularities, successfully applying it to prune weights, filters, subnetworks in convolutional architectures and nodes in recurrent networks.
Pruning is one of the most effective model reduction techniques. Deep networks require massive computation and such models need to be compressed to bring them on edge devices. Most existing pruning techniques are focused on vision-based models like convolutional networks, while text-based models are still evolving. The emergence of multi-modal multi-task learning calls for a general method that works on vision and text architectures simultaneously. We introduce a \emph{differentiable mask}, that induces sparsity on various granularity to fill this gap. We apply our method successfully to prune weights, filters, subnetwork of a convolutional architecture, as well as nodes of a recurrent network.