A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
This work addresses the optimization bottleneck in BNNs, which are crucial for efficient deployment on resource-constrained devices, though it represents an incremental improvement over the existing Bop optimizer.
The paper tackles the problem of directly training Binary Neural Networks (BNNs) by proposing Bop2ndOrder, a second-order optimizer that uses second raw moment estimates to normalize gradients before deciding weight bit-flips. The method achieved faster convergence, better robustness to hyperparameter changes, and higher accuracy on CIFAR10 and ImageNet datasets compared to existing approaches.
The optimization of Binary Neural Networks (BNNs) relies on approximating the real-valued weights with their binarized representations. Current techniques for weight-updating use the same approaches as traditional Neural Networks (NNs) with the extra requirement of using an approximation to the derivative of the sign function - as it is the Dirac-Delta function - for back-propagation; thus, efforts are focused adapting full-precision techniques to work on BNNs. In the literature, only one previous effort has tackled the problem of directly training the BNNs with bit-flips by using the first raw moment estimate of the gradients and comparing it against a threshold for deciding when to flip a weight (Bop). In this paper, we take an approach parallel to Adam which also uses the second raw moment estimate to normalize the first raw moment before doing the comparison with the threshold, we call this method Bop2ndOrder. We present two versions of the proposed optimizer: a biased one and a bias-corrected one, each with its own applications. Also, we present a complete ablation study of the hyperparameters space, as well as the effect of using schedulers on each of them. For these studies, we tested the optimizer in CIFAR10 using the BinaryNet architecture. Also, we tested it in ImageNet 2012 with the XnorNet and BiRealNet architectures for accuracy. In both datasets our approach proved to converge faster, was robust to changes of the hyperparameters, and achieved better accuracy values.