Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
This work addresses the computational inefficiency of BN for practitioners in deep learning, but it is incremental as it confirms BN's superiority in large-scale settings.
The study compared batch normalization (BN) and weight normalization (WN) algorithms for large-scale image classification using ResNet-50 on ImageNet, finding that WN achieved better training accuracy but had about 6% lower test accuracy and less stable training than BN.
Batch normalization (BN) has become a de facto standard for training deep convolutional networks. However, BN accounts for a significant fraction of training run-time and is difficult to accelerate, since it is a memory-bandwidth bounded operation. Such a drawback of BN motivates us to explore recently proposed weight normalization algorithms (WN algorithms), i.e. weight normalization, normalization propagation and weight normalization with translated ReLU. These algorithms don't slow-down training iterations and were experimentally shown to outperform BN on relatively small networks and datasets. However, it is not clear if these algorithms could replace BN in practical, large-scale applications. We answer this question by providing a detailed comparison of BN and WN algorithms using ResNet-50 network trained on ImageNet. We found that although WN achieves better training accuracy, the final test accuracy is significantly lower ($\approx 6\%$) than that of BN. This result demonstrates the surprising strength of the BN regularization effect which we were unable to compensate for using standard regularization techniques like dropout and weight decay. We also found that training of deep networks with WN algorithms is significantly less stable compared to BN, limiting their practical applications.