Regularizing Activation Distribution for Training Binarized Deep Networks
This work addresses the problem of efficient and accurate training for BNNs, which is crucial for deploying neural networks on resource-constrained devices, representing an incremental improvement over prior methods that compromised energy efficiency.
The paper tackles the difficulty of training Binarized Neural Networks (BNNs) due to activation flow issues like degeneration and gradient mismatch, and shows that using a distribution loss to regularize activations improves accuracy without sacrificing energy efficiency, with experiments demonstrating consistent gains and robustness to hyper-parameter choices.
Binarized Neural Networks (BNNs) can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN's energy efficiency. In this paper, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.