Lipschitz Continuity Retained Binary Neural Network
This work addresses robustness issues in BNNs for efficient deep learning applications, representing an incremental improvement with a novel method for a known bottleneck.
The authors tackled the problem of improving robustness in binary neural networks (BNNs) by introducing Lipschitz continuity as a rigorous criterion and proposing a regularization method using Retention Matrices to approximate spectral norms, achieving state-of-the-art performance on CIFAR and ImageNet datasets.
Relying on the premise that the performance of a binary neural network can be largely restored with eliminated quantization error between full-precision weight vectors and their corresponding binary vectors, existing works of network binarization frequently adopt the idea of model robustness to reach the aforementioned objective. However, robustness remains to be an ill-defined concept without solid theoretical support. In this work, we introduce the Lipschitz continuity, a well-defined functional property, as the rigorous criteria to define the model robustness for BNN. We then propose to retain the Lipschitz continuity as a regularization term to improve the model robustness. Particularly, while the popular Lipschitz-involved regularization methods often collapse in BNN due to its extreme sparsity, we design the Retention Matrices to approximate spectral norms of the targeted weight matrices, which can be deployed as the approximation for the Lipschitz constant of BNNs without the exact Lipschitz constant computation (NP-hard). Our experiments prove that our BNN-specific regularization method can effectively strengthen the robustness of BNN (testified on ImageNet-C), achieving state-of-the-art performance on CIFAR and ImageNet.