Controlling Covariate Shift using Balanced Normalization of Weights
This work addresses covariate shift for deep learning practitioners by offering an incremental improvement over existing normalization methods.
The paper tackles the problem of covariate shift in neural networks by introducing a new normalization technique that balances positive and negative weights to maintain fast convergence similar to batch normalization, achieving competitive results on standard benchmarks like CIFAR-10/100, SVHN, and ImageNet.
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.