Accelerated Stochastic Quasi-Newton Optimization on Riemann Manifolds
This work addresses optimization challenges in machine learning and data analysis for problems involving Riemannian manifolds, representing an incremental improvement by combining existing techniques like L-BFGS and stochastic variance reduction.
The paper tackles optimization on Riemann manifolds by proposing an L-BFGS algorithm with stochastic variance reduction for fast convergence using constant step sizes, showing strong convergence results in experiments on Karcher means and leading eigenvalues compared to methods like VR-PCA and Riemannian SVRG.
We propose an L-BFGS optimization algorithm on Riemannian manifolds using minibatched stochastic variance reduction techniques for fast convergence with constant step sizes, without resorting to linesearch methods designed to satisfy Wolfe conditions. We provide a new convergence proof for strongly convex functions without using curvature conditions on the manifold, as well as a convergence discussion for nonconvex functions. We discuss a couple of ways to obtain the correction pairs used to calculate the product of the gradient with the inverse Hessian, and empirically demonstrate their use in synthetic experiments on computation of Karcher means for symmetric positive definite matrices and leading eigenvalues of large scale data matrices. We compare our method to VR-PCA for the latter experiment, along with Riemannian SVRG for both cases, and show strong convergence results for a range of datasets.