A Neural Network model with Bidirectional Whitening
This work addresses optimization challenges in neural networks for researchers, but it is incremental as it builds on existing whitening methods.
The authors tackled the problem of improving natural gradient descent for multilayer perceptrons by extending whitening to both feed-forward and back-propagation phases, resulting in a 'Bidirectional whitened neural networks' model that demonstrated efficacy on the MNIST handwritten character recognition dataset.
We present here a new model and algorithm which performs an efficient Natural gradient descent for Multilayer Perceptrons. Natural gradient descent was originally proposed from a point of view of information geometry, and it performs the steepest descent updates on manifolds in a Riemannian space. In particular, we extend an approach taken by the "Whitened neural networks" model. We make the whitening process not only in feed-forward direction as in the original model, but also in the back-propagation phase. Its efficacy is shown by an application of this "Bidirectional whitened neural networks" model to a handwritten character recognition data (MNIST data).