Data-Driven Low-Rank Neural Network Compression
This addresses the storage and inference efficiency problem for deploying neural networks on resource-limited devices, representing an incremental improvement over existing compression techniques.
The paper tackles the problem of compressing deep neural networks for deployment on storage-constrained devices by proposing a Data-Driven Low-rank (DDLR) method that reduces parameters in fully connected layers without retraining, achieving significant compression with only a small accuracy reduction and outperforming the sparsity-based Net-Trim method in compression and accuracy.
Despite many modern applications of Deep Neural Networks (DNNs), the large number of parameters in the hidden layers makes them unattractive for deployment on devices with storage capacity constraints. In this paper we propose a Data-Driven Low-rank (DDLR) method to reduce the number of parameters of pretrained DNNs and expedite inference by imposing low-rank structure on the fully connected layers, while controlling for the overall accuracy and without requiring any retraining. We pose the problem as finding the lowest rank approximation of each fully connected layer with given performance guarantees and relax it to a tractable convex optimization problem. We show that it is possible to significantly reduce the number of parameters in common DNN architectures with only a small reduction in classification accuracy. We compare DDLR with Net-Trim, which is another data-driven DNN compression technique based on sparsity and show that DDLR consistently produces more compressed neural networks while maintaining higher accuracy.