Riemannian stochastic variance reduced gradient on Grassmann manifold
This work addresses optimization challenges on manifolds for researchers in machine learning and data analysis, representing an incremental improvement over existing methods.
The authors tackled the problem of optimizing functions on Grassmann manifolds by proposing a Riemannian stochastic variance reduced gradient algorithm, which outperformed standard Riemannian stochastic gradient descent in applications like principal components analysis and low-rank matrix completion.
Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite, number of loss functions. In this paper, we propose a novel Riemannian extension of the Euclidean stochastic variance reduced gradient algorithm (R-SVRG) to a compact manifold search space. To this end, we show the developments on the Grassmann manifold. The key challenges of averaging, addition, and subtraction of multiple gradients are addressed with notions like logarithm mapping and parallel translation of vectors on the Grassmann manifold. We present a global convergence analysis of the proposed algorithm with decay step-sizes and a local convergence rate analysis under fixed step-size with some natural assumptions. The proposed algorithm is applied on a number of problems on the Grassmann manifold like principal components analysis, low-rank matrix completion, and the Karcher mean computation. In all these cases, the proposed algorithm outperforms the standard Riemannian stochastic gradient descent algorithm.