Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold
This work addresses nonsmooth optimization for machine learning applications like sparse PCA and matrix completion, representing an incremental advancement by extending Euclidean proximal methods to Riemannian manifolds.
The authors tackled the problem of nonsmooth optimization over the Stiefel manifold by proposing two Riemannian stochastic proximal gradient methods, R-ProxSGD and R-ProxSPB, which achieve improved performance with R-ProxSPB finding an ε-stationary point with Ø(ε^{-3}) IFOs in the online case and Ø(n+√n ε^{-2}) in the finite-sum case.
Riemannian optimization has drawn a lot of attention due to its wide applications in practice. Riemannian stochastic first-order algorithms have been studied in the literature to solve large-scale machine learning problems over Riemannian manifolds. However, most of the existing Riemannian stochastic algorithms require the objective function to be differentiable, and they do not apply to the case where the objective function is nonsmooth. In this paper, we present two Riemannian stochastic proximal gradient methods for minimizing nonsmooth function over the Stiefel manifold. The two methods, named R-ProxSGD and R-ProxSPB, are generalizations of proximal SGD and proximal SpiderBoost in Euclidean setting to the Riemannian setting. Analysis on the incremental first-order oracle (IFO) complexity of the proposed algorithms is provided. Specifically, the R-ProxSPB algorithm finds an $ε$-stationary point with $Ø(ε^{-3})$ IFOs in the online case, and $Ø(n+\sqrt{n}ε^{-2})$ IFOs in the finite-sum case with $n$ being the number of summands in the objective. Experimental results on online sparse PCA and robust low-rank matrix completion show that our proposed methods significantly outperform the existing methods that use Riemannian subgradient information.