An Empirical Study on Position of the Batch Normalization Layer in Convolutional Neural Networks
This work addresses optimization of training efficiency for deep learning practitioners, but it is incremental as it builds on existing batch normalization techniques.
The study investigated how changing the position of the batch normalization layer affects training speed in convolutional neural networks, finding that alternative positions can improve training speed compared to the original suggestion, with specific impacts varying across AlexNet, VGG-16, and ResNet-20.
In this paper, we have studied how the training of the convolutional neural networks (CNNs) can be affected by changing the position of the batch normalization (BN) layer. Three different convolutional neural networks have been chosen for our experiments. These networks are AlexNet, VGG-16, and ResNet- 20. We show that the speed up in training provided by the BN algorithm can be improved by using other positions for the BN layer than the one suggested by its original paper. Also, we discuss how the BN layer in a certain position can aid the training of one network but not the other. Three different positions for the BN layer have been studied in this research. These positions are: the BN layer between the convolution layer and the non-linear activation function, the BN layer after the non-linear activation function and finally, the BN layer before each of the convolutional layers.