BiPer: Binary Neural Networks using a Periodic Function
This work addresses the problem of efficient neural network deployment for resource-constrained devices by improving BNN performance, though it is incremental as it builds on existing binarization techniques.
The paper tackles the training inconsistency and performance degradation in Binary Neural Networks (BNNs) by proposing a binary periodic (BiPer) function, which uses a square wave for the forward pass and a sine function for the backward pass, resulting in improvements of up to 1% on CIFAR-10 and 0.69% on ImageNet compared to state-of-the-art methods.
Quantized neural networks employ reduced precision representations for both weights and activations. This quantization process significantly reduces the memory requirements and computational complexity of the network. Binary Neural Networks (BNNs) are the extreme quantization case, representing values with just one bit. Since the sign function is typically used to map real values to binary values, smooth approximations are introduced to mimic the gradients during error backpropagation. Thus, the mismatch between the forward and backward models corrupts the direction of the gradient, causing training inconsistency problems and performance degradation. In contrast to current BNN approaches, we propose to employ a binary periodic (BiPer) function during binarization. Specifically, we use a square wave for the forward pass to obtain the binary values and employ the trigonometric sine function with the same period of the square wave as a differentiable surrogate during the backward pass. We demonstrate that this approach can control the quantization error by using the frequency of the periodic function and improves network performance. Extensive experiments validate the effectiveness of BiPer in benchmark datasets and network architectures, with improvements of up to 1% and 0.69% with respect to state-of-the-art methods in the classification task over CIFAR-10 and ImageNet, respectively. Our code is publicly available at https://github.com/edmav4/BiPer.