Riemannian stochastic recursive momentum method for non-convex optimization
This work addresses optimization challenges in machine learning and related fields by improving efficiency for non-convex problems on manifolds, representing an incremental advancement over existing methods.
The paper tackles non-convex optimization on Riemannian manifolds by proposing a stochastic recursive momentum method that achieves a near-optimal complexity of Õ(ε⁻³) to find an ε-approximate solution with one sample per iteration, eliminating the need for large batch gradients.
We propose a stochastic recursive momentum method for Riemannian non-convex optimization that achieves a near-optimal complexity of $\tilde{\mathcal{O}}(ε^{-3})$ to find $ε$-approximate solution with one sample. That is, our method requires $\mathcal{O}(1)$ gradient evaluations per iteration and does not require restarting with a large batch gradient, which is commonly used to obtain the faster rate. Extensive experiment results demonstrate the superiority of our proposed algorithm.