How Does Batch Normalization Help Binary Training?
This addresses training instability in BNNs for efficient deep learning applications, but it is incremental as it builds on existing BatchNorm insights.
The paper tackled the problem of training Binary Neural Networks (BNNs), which often fail without Batch Normalization, by finding that BatchNorm prevents exploding gradients, making common initialization methods irrelevant for BNNs.
Binary Neural Networks (BNNs) are difficult to train, and suffer from drop of accuracy. It appears in practice that BNNs fail to train in the absence of Batch Normalization (BatchNorm) layer. We find the main role of BatchNorm is to avoid exploding gradients in the case of BNNs. This finding suggests that the common initialization methods developed for full-precision networks are irrelevant to BNNs. We build a theoretical study on the role of BatchNorm in binary training, backed up by numerical experiments.