Pushing the Efficiency Limit Using Structured Sparse Convolutions
This addresses the efficiency bottleneck in deploying deep neural networks, particularly for resource-constrained applications, by providing a more effective compression method compared to existing pruning techniques.
The paper tackles the problem of computationally expensive weight pruning for compressing deep convolutional neural networks by proposing Structured Sparse Convolution (SSC), which leverages image structure to reduce parameters and achieves state-of-the-art performance on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet benchmarks.
Weight pruning is among the most popular approaches for compressing deep convolutional neural networks. Recent work suggests that in a randomly initialized deep neural network, there exist sparse subnetworks that achieve performance comparable to the original network. Unfortunately, finding these subnetworks involves iterative stages of training and pruning, which can be computationally expensive. We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter. This leads to improved efficiency of convolutional architectures compared to existing methods that perform pruning at initialization. We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in ``efficient architectures.'' Extensive experiments on well-known CNN models and datasets show the effectiveness of the proposed method. Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.