Practical Riemannian Neural Networks
This work provides practical implementation and experimental validation for Riemannian neural networks, which is incremental as it builds on prior theoretical work.
The authors tackled the problem of applying quasi-diagonal Riemannian gradient descents to neural networks on non-synthetic datasets, showing that these methods consistently outperform simple stochastic gradient descent with a computational overhead of about 2x and achieve faster convergence, requiring fewer training epochs and less total computation time.
We provide the first experimental results on non-synthetic datasets for the quasi-diagonal Riemannian gradient descents for neural networks introduced in [Ollivier, 2015]. These include the MNIST, SVHN, and FACE datasets as well as a previously unpublished electroencephalogram dataset. The quasi-diagonal Riemannian algorithms consistently beat simple stochastic gradient gradient descents by a varying margin. The computational overhead with respect to simple backpropagation is around a factor $2$. Perhaps more interestingly, these methods also reach their final performance quickly, thus requiring fewer training epochs and a smaller total computation time. We also present an implementation guide to these Riemannian gradient descents for neural networks, showing how the quasi-diagonal versions can be implemented with minimal effort on top of existing routines which compute gradients.