Random Bias Initialization Improves Quantized Training
This work addresses the accuracy drop in binary neural networks, which is an incremental improvement for efficient deep learning models.
The paper tackles the performance gap between full-precision and binary neural networks by analyzing their geometry and proposing random bias initialization as a counter-intuitive remedy to improve quantized training.
Binary neural networks improve computationally efficiency of deep models with a large margin. However, there is still a performance gap between a successful full-precision training and binary training. We bring some insights about why this accuracy drop exists and call for a better understanding of binary network geometry. We start with analyzing full-precision neural networks with ReLU activation and compare it with its binarized version. This comparison suggests to initialize networks with random bias, a counter-intuitive remedy.