Mean-field Analysis of Batch Normalization
This provides theoretical justification for a widely used technique in deep learning, offering incremental insights into optimization improvements for practitioners.
The paper tackled the problem of understanding how Batch Normalization affects neural network training by using mean-field theory to show it flattens the loss landscape, reducing the maximum eigenvalue of the Fisher Information Matrix by up to 50%, which justifies using larger learning rates and leads to faster convergence.
Batch Normalization (BatchNorm) is an extremely useful component of modern neural network architectures, enabling optimization using higher learning rates and achieving faster convergence. In this paper, we use mean-field theory to analytically quantify the impact of BatchNorm on the geometry of the loss landscape for multi-layer networks consisting of fully-connected and convolutional layers. We show that it has a flattening effect on the loss landscape, as quantified by the maximum eigenvalue of the Fisher Information Matrix. These findings are then used to justify the use of larger learning rates for networks that use BatchNorm, and we provide quantitative characterization of the maximal allowable learning rate to ensure convergence. Experiments support our theoretically predicted maximum learning rate, and furthermore suggest that networks with smaller values of the BatchNorm parameter achieve lower loss after the same number of epochs of training.