Controlling Information Capacity of Binary Neural Network
This work addresses the limited accuracy of binary networks for mobile and IoT applications, representing an incremental improvement.
The paper tackles the accuracy degradation problem in binary neural networks by introducing a training method that maintains a predefined information capacity using Shannon entropy penalties, resulting in statistically significant accuracy improvements on SVHN, CIFAR, and ImageNet datasets.
Despite the growing popularity of deep learning technologies, high memory requirements and power consumption are essentially limiting their application in mobile and IoT areas. While binary convolutional networks can alleviate these problems, the limited bitwidth of weights is often leading to significant degradation of prediction accuracy. In this paper, we present a method for training binary networks that maintains a stable predefined level of their information capacity throughout the training process by applying Shannon entropy based penalty to convolutional filters. The results of experiments conducted on SVHN, CIFAR and ImageNet datasets demonstrate that the proposed approach can statistically significantly improve the accuracy of binary networks.