Low-Rank Embedding of Kernels in Convolutional Neural Networks under Random Shuffling
This addresses storage reduction for resource-limited platforms in image processing, but it is incremental as it builds on existing tensor decomposition methods.
The paper tackles the problem of reducing storage costs in CNNs by proposing a method to embed convolutional kernels into random low-rank subspaces via randomly-shuffled tensor decomposition, achieving significant compression and more stable accuracy across compression ratios on CIFAR-10.
Although the convolutional neural networks (CNNs) have become popular for various image processing and computer vision task recently, it remains a challenging problem to reduce the storage cost of the parameters for resource-limited platforms. In the previous studies, tensor decomposition (TD) has achieved promising compression performance by embedding the kernel of a convolutional layer into a low-rank subspace. However the employment of TD is naively on the kernel or its specified variants. Unlike the conventional approaches, this paper shows that the kernel can be embedded into more general or even random low-rank subspaces. We demonstrate this by compressing the convolutional layers via randomly-shuffled tensor decomposition (RsTD) for a standard classification task using CIFAR-10. In addition, we analyze how the spatial similarity of the training data influences the low-rank structure of the kernels. The experimental results show that the CNN can be significantly compressed even if the kernels are randomly shuffled. Furthermore, the RsTD-based method yields more stable classification accuracy than the conventional TD-based methods in a large range of compression ratios.