Accurate and Compact Convolutional Neural Networks with Trained Binarization
This work addresses the deployment of CNNs on mobile and embedded devices by reducing resource demands, though it is incremental as it builds on existing binary CNN methods.
The paper tackles the problem of accuracy degradation in binary convolutional neural networks (CNNs) by proposing an improved training approach with trainable scaling factors and a specific algorithm, achieving 92.3% accuracy on CIFAR-10 with VGG-Small and 46.1% top-1 accuracy on ImageNet with AlexNet.
Although convolutional neural networks (CNNs) are now widely used in various computer vision applications, its huge resource demanding on parameter storage and computation makes the deployment on mobile and embedded devices difficult. Recently, binary convolutional neural networks are explored to help alleviate this issue by quantizing both weights and activations with only 1 single bit. However, there may exist a noticeable accuracy degradation when compared with full-precision models. In this paper, we propose an improved training approach towards compact binary CNNs with higher accuracy. Trainable scaling factors for both weights and activations are introduced to increase the value range. These scaling factors will be trained jointly with other parameters via backpropagation. Besides, a specific training algorithm is developed including tight approximation for derivative of discontinuous binarization function and $L_2$ regularization acting on weight scaling factors. With these improvements, the binary CNN achieves 92.3% accuracy on CIFAR-10 with VGG-Small network. On ImageNet, our method also obtains 46.1% top-1 accuracy with AlexNet and 54.2% with Resnet-18 surpassing previous works.