Probabilistic Binary Neural Networks
This work addresses the need for efficient deep neural networks for deployment in resource-constrained environments, though it is incremental in improving binary network training.
The authors tackled the challenge of training binary neural networks with both binary weights and activations by introducing a probabilistic training method called BLRNet, which avoids gradient approximation issues and achieves competitive performance on standardized benchmarks.
Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called BLRNet. By embracing stochasticity during training, we circumvent the need to approximate the gradient of non-differentiable functions such as sign(), while still obtaining a fully Binary Neural Network at test time. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution. Since all operations in a layer of the BLRNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time. We evaluate the BLRNet on multiple standardized benchmarks.