Network Sketching: Exploiting Binary Structure in Deep CNNs
This work addresses the challenge of deploying efficient deep CNNs on mobile devices by improving the accuracy of binary-weight models, though it is incremental as it builds on existing binary-weight approaches.
The paper tackled the problem of binary-weight CNNs having unsatisfactory accuracy for practical applications by introducing network sketching, a technique that exploits binary structure in pre-trained filters to produce models with better trade-offs, resulting in sketches of AlexNet and ResNet outperforming existing binary-weight models by large margins on ImageNet classification while only slightly increasing memory usage.
Convolutional neural networks (CNNs) with deep architectures have substantially advanced the state-of-the-art in computer vision tasks. However, deep networks are typically resource-intensive and thus difficult to be deployed on mobile devices. Recently, CNNs with binary weights have shown compelling efficiency to the community, whereas the accuracy of such models is usually unsatisfactory in practice. In this paper, we introduce network sketching as a novel technique of pursuing binary-weight CNNs, targeting at more faithful inference and better trade-off for practical applications. Our basic idea is to exploit binary structure directly in pre-trained filter banks and produce binary-weight models via tensor expansion. The whole process can be treated as a coarse-to-fine model approximation, akin to the pencil drawing steps of outlining and shading. To further speedup the generated models, namely the sketches, we also propose an associative implementation of binary tensor convolutions. Experimental results demonstrate that a proper sketch of AlexNet (or ResNet) outperforms the existing binary-weight models by large margins on the ImageNet large scale classification task, while the committed memory for network parameters only exceeds a little.