Batch normalization does not improve initialization
This addresses a theoretical gap in understanding batch normalization for neural network researchers, but it is incremental as it refutes a specific prior claim.
The paper challenges the claim that batch normalization improves initialization by providing a counterexample, showing it does not enhance initialization as previously asserted.
Batch normalization is one of the most important regularization techniques for neural networks, significantly improving training by centering the layers of the neural network. There have been several attempts to provide a theoretical justification for batch ormalization. Santurkar and Tsipras (2018) [How does batch normalization help optimization? Advances in neural information rocessing systems, 31] claim that batch normalization improves initialization. We provide a counterexample showing that this claim s not true, i.e., batch normalization does not improve initialization.