Widening and Squeezing: Towards Accurate and Efficient QNNs
This addresses the need for efficient and accurate neural networks for industry applications, but it is incremental as it builds on existing quantization methods.
The paper tackles the problem of quantization neural networks (QNNs) having weaker performance than full-precision networks by projecting features to high-dimensional quantization features and eliminating redundancy, resulting in QNNs with much fewer parameters and calculations while achieving nearly the same performance, e.g., 29.9% top-1 error for binary ResNet-18 on ImageNet.
Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters. Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques. However, we find the representation capability of quantization features is far weaker than full-precision features by experiments. We address this problem by projecting features in original full-precision networks to high-dimensional quantization features. Simultaneously, redundant quantization features will be eliminated in order to avoid unrestricted growth of dimensions for some datasets. Then, a compact quantization neural network but with sufficient representation ability will be established. Experimental results on benchmark datasets demonstrate that the proposed method is able to establish QNNs with much less parameters and calculations but almost the same performance as that of full-precision baseline models, e.g. $29.9\%$ top-1 error of binary ResNet-18 on the ImageNet ILSVRC 2012 dataset.