Regularized Binary Network Training
This work addresses efficiency and accuracy challenges in deploying neural networks on resource-constrained devices, representing an incremental improvement in binary network training.
The paper tackles the performance gap between Binary Neural Networks (BNNs) and floating-point DNNs by introducing a regularization function with trainable scaling factors and an improved derivative approximation for sign activation, resulting in experimental improvements on ImageNet over traditional BNN and XNOR-net methods.
There is a significant performance gap between Binary Neural Networks (BNNs) and floating point Deep Neural Networks (DNNs). We propose to improve the binary training method, by introducing a new regularization function that encourages training weights around binary values. In addition, we add trainable scaling factors to our regularization functions. Additionally, an improved approximation of the derivative of the sign activation function in the backward computation. These modifications are based on linear operations that are easily implementable into the binary training framework. Experimental results on ImageNet shows our method outperforms the traditional BNN method and XNOR-net.