Entropy-Constrained Training of Deep Neural Networks
This provides a general compression method for deep learning models, though it appears incremental as it unifies existing techniques under a theoretical framework.
The authors tackled neural network compression by proposing an entropy-constrained optimization framework based on the Minimum Description Length principle, achieving state-of-the-art results such as 71x compression on a VGG-like architecture.
We propose a general framework for neural network compression that is motivated by the Minimum Description Length (MDL) principle. For that we first derive an expression for the entropy of a neural network, which measures its complexity explicitly in terms of its bit-size. Then, we formalize the problem of neural network compression as an entropy-constrained optimization objective. This objective generalizes many of the compression techniques proposed in the literature, in that pruning or reducing the cardinality of the weight elements of the network can be seen special cases of entropy-minimization techniques. Furthermore, we derive a continuous relaxation of the objective, which allows us to minimize it using gradient based optimization techniques. Finally, we show that we can reach state-of-the-art compression results on different network architectures and data sets, e.g. achieving x71 compression gains on a VGG-like architecture.