Riemannian stochastic quasi-Newton algorithm with variance reduction and its convergence analysis
This work addresses optimization challenges on Riemannian manifolds for applications in machine learning and data analysis, representing an incremental improvement over existing methods.
The paper tackles the problem of minimizing the average of many loss functions on Riemannian manifolds by proposing a Riemannian stochastic quasi-Newton algorithm with variance reduction (R-SQN-VR), which outperforms state-of-the-art Riemannian batch and stochastic gradient algorithms in tasks like Karcher mean computation and low-rank matrix completion.
Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite number of loss functions. The present paper proposes a Riemannian stochastic quasi-Newton algorithm with variance reduction (R-SQN-VR). The key challenges of averaging, adding, and subtracting multiple gradients are addressed with notions of retraction and vector transport. We present convergence analyses of R-SQN-VR on both non-convex and retraction-convex functions under retraction and vector transport operators. The proposed algorithm is evaluated on the Karcher mean computation on the symmetric positive-definite manifold and the low-rank matrix completion on the Grassmann manifold. In all cases, the proposed algorithm outperforms the state-of-the-art Riemannian batch and stochastic gradient algorithms.