Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression
This addresses the challenge of deploying large CNNs in resource-limited environments, offering an incremental improvement over existing decomposition-based compression techniques.
The paper tackles the problem of compressing deep convolutional neural networks (CNNs) for deployment on resource-constrained platforms by proposing joint matrix decomposition to reduce performance degradation, achieving a 22X compression on ResNet-34 with slighter accuracy loss compared to state-of-the-art methods.
Deep convolutional neural networks (CNNs) with a large number of parameters require intensive computational resources, and thus are hard to be deployed in resource-constrained platforms. Decomposition-based methods, therefore, have been utilized to compress CNNs in recent years. However, since the compression factor and performance are negatively correlated, the state-of-the-art works either suffer from severe performance degradation or have relatively low compression factors. To overcome this problem, we propose to compress CNNs and alleviate performance degradation via joint matrix decomposition, which is different from existing works that compressed layers separately. The idea is inspired by the fact that there are lots of repeated modules in CNNs. By projecting weights with the same structures into the same subspace, networks can be jointly compressed with larger ranks. In particular, three joint matrix decomposition schemes are developed, and the corresponding optimization approaches based on Singular Value Decomposition are proposed. Extensive experiments are conducted across three challenging compact CNNs for different benchmark data sets to demonstrate the superior performance of our proposed algorithms. As a result, our methods can compress the size of ResNet-34 by 22X with slighter accuracy degradation compared with several state-of-the-art methods.