Wide Compression: Tensor Ring Nets
This addresses memory and computation limitations for deploying deep learning models on resource-constrained devices like smartphones and IoT, though it is incremental as it builds on existing tensor ring factorization.
The paper tackles the problem of large memory and computational requirements of deep neural networks by introducing Tensor Ring Networks (TR-Nets), which compress models significantly, achieving up to 243x compression on Wide ResNet with only 2.3% accuracy degradation on Cifar10.
Deep neural networks have demonstrated state-of-the-art performance in a variety of real-world applications. In order to obtain performance gains, these networks have grown larger and deeper, containing millions or even billions of parameters and over a thousand layers. The trade-off is that these large architectures require an enormous amount of memory, storage, and computation, thus limiting their usability. Inspired by the recent tensor ring factorization, we introduce Tensor Ring Networks (TR-Nets), which significantly compress both the fully connected layers and the convolutional layers of deep neural networks. Our results show that our TR-Nets approach {is able to compress LeNet-5 by $11\times$ without losing accuracy}, and can compress the state-of-the-art Wide ResNet by $243\times$ with only 2.3\% degradation in {Cifar10 image classification}. Overall, this compression scheme shows promise in scientific computing and deep learning, especially for emerging resource-constrained devices such as smartphones, wearables, and IoT devices.