An Adaptive Tensor-Train Decomposition Approach for Efficient Deep Neural Network Compression
This addresses the computational burden and inefficiency in model compression for deep learning practitioners, though it is incremental as it builds on existing tensor decomposition methods.
The paper tackles the problem of inefficient rank selection in tensor decomposition for neural network compression by introducing a budget-aware automatic method called Layer-Wise Imprinting Quantitation (LWIQ), which improves rank search efficiency by 63.2% and reduces model size by 3.2x with only a 0.86% accuracy drop on ResNet-56.
In the field of model compression, choosing an appropriate rank for tensor decomposition is pivotal for balancing model compression rate and efficiency. However, this selection, whether done manually or through optimization-based automatic methods, often increases computational complexity. Manual rank selection lacks efficiency and scalability, often requiring extensive trial-and-error, while optimization-based automatic methods significantly increase the computational burden. To address this, we introduce a novel, automatic, and budget-aware rank selection method for efficient model compression, which employs Layer-Wise Imprinting Quantitation (LWIQ). LWIQ quantifies each layer's significance within a neural network by integrating a proxy classifier. This classifier assesses the layer's impact on overall model performance, allowing for a more informed adjustment of tensor rank. Furthermore, our approach includes a scaling factor to cater to varying computational budget constraints. This budget awareness eliminates the need for repetitive rank recalculations for different budget scenarios. Experimental results on the CIFAR-10 dataset show that our LWIQ improved by 63.2% in rank search efficiency, and the accuracy only dropped by 0.86% with 3.2x less model size on the ResNet-56 model as compared to the state-of-the-art proxy-based automatic tensor rank selection method.