A Theoretical Understanding of Neural Network Compression from Sparse Linear Approximation
This work addresses the need for theoretical foundations in model compression for resource-limited applications, though it appears incremental as it builds on existing pruning algorithms with new theoretical insights.
The authors tackled the problem of theoretically understanding neural network compressibility by proposing a sparsity-sensitive norm to characterize compressibility and developing adaptive pruning algorithms, with numerical studies showing promising performance compared to standard methods.
The goal of model compression is to reduce the size of a large neural network while retaining a comparable performance. As a result, computation and memory costs in resource-limited applications may be significantly reduced by dropping redundant weights, neurons, or layers. There have been many model compression algorithms proposed that provide impressive empirical success. However, a theoretical understanding of model compression is still limited. One problem is understanding if a network is more compressible than another of the same structure. Another problem is quantifying how much one can prune a network with theoretically guaranteed accuracy degradation. In this work, we propose to use the sparsity-sensitive $\ell_q$-norm ($0<q<1$) to characterize compressibility and provide a relationship between soft sparsity of the weights in the network and the degree of compression with a controlled accuracy degradation bound. We also develop adaptive algorithms for pruning each neuron in the network informed by our theory. Numerical studies demonstrate the promising performance of the proposed methods compared with standard pruning algorithms.