Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
This is an incremental contribution that provides a taxonomy and evaluation framework for researchers and practitioners working on neural network compression.
The paper tackles the problem of making convolutional neural networks more efficient through compression, providing a new categorization of methods based on data/compute requirements and showing that combining SVD with probabilistic compression/pruning yields the best results on ImageNet.
The success of deep neural networks in many real-world applications is leading to new challenges in building more efficient architectures. One effective way of making networks more efficient is neural network compression. We provide an overview of existing neural network compression methods that can be used to make neural networks more efficient by changing the architecture of the network. First, we introduce a new way to categorize all published compression methods, based on the amount of data and compute needed to make the methods work in practice. These are three 'levels of compression solutions'. Second, we provide a taxonomy of tensor factorization based and probabilistic compression methods. Finally, we perform an extensive evaluation of different compression techniques from the literature for models trained on ImageNet. We show that SVD and probabilistic compression or pruning methods are complementary and give the best results of all the considered methods. We also provide practical ways to combine them.