Batch Normalization in Quantized Networks
This addresses a specific issue in quantized network training, which is incremental as it builds on prior observations to clarify BatchNorm's effects.
The paper tackles the problem of understanding batch normalization's role in training quantized neural networks, showing that it prevents gradient explosion, a counter-intuitive effect recently observed in numerical experiments.
Implementation of quantized neural networks on computing hardware leads to considerable speed up and memory saving. However, quantized deep networks are difficult to train and batch~normalization (BatchNorm) layer plays an important role in training full-precision and quantized networks. Most studies on BatchNorm are focused on full-precision networks, and there is little research in understanding BatchNorm affect in quantized training which we address here. We show BatchNorm avoids gradient explosion which is counter-intuitive and recently observed in numerical experiments by other researchers.