Make $\ell_1$ Regularization Effective in Training Sparse CNN
This addresses the challenge of model compression for deep learning practitioners by enabling efficient sparsification in CNNs, though it is incremental in adapting existing convex methods to non-convex settings.
The paper tackled the problem of making $\ell_1$ regularization effective for training sparse convolutional neural networks (CNNs), achieving state-of-the-art sparsity levels, such as 95% for ResNet18 on CIFAR-10, without compromising accuracy.
Compressed Sensing using $\ell_1$ regularization is among the most powerful and popular sparsification technique in many applications, but why has it not been used to obtain sparse deep learning model such as convolutional neural network (CNN)? This paper is aimed to provide an answer to this question and to show how to make it work. We first demonstrate that the commonly used stochastic gradient decent (SGD) and variants training algorithm is not an appropriate match with $\ell_1$ regularization and then replace it with a different training algorithm based on a regularized dual averaging (RDA) method. RDA was originally designed specifically for convex problem, but with new theoretical insight and algorithmic modifications (using proper initialization and adaptivity), we have made it an effective match with $\ell_1$ regularization to achieve a state-of-the-art sparsity for CNN compared to other weight pruning methods without compromising accuracy (achieving 95\% sparsity for ResNet18 on CIFAR-10, for example).