Operator-valued formulas for Riemannian Gradient and Hessian and families of tractable metrics
This work extends the toolkit for manifold optimization in machine learning by making more metrics tractable, though it is incremental in advancing existing mathematical frameworks.
The authors tackled the problem of computing Riemannian gradients and Hessians on quotient manifolds with non-constant metrics, providing explicit formulas that enable Riemannian optimization on classical manifolds like Stiefel and flag manifolds, and introducing new metrics on positive-semidefinite matrices of fixed rank.
We provide an explicit formula for the Levi-Civita connection and Riemannian Hessian for a Riemannian manifold that is a quotient of a manifold embedded in an inner product space with a non-constant metric function. Together with a classical formula for projection, this allows us to evaluate Riemannian gradient and Hessian for several families of metrics on classical manifolds, including a family of metrics on Stiefel manifolds connecting both the constant and canonical ambient metrics with closed-form geodesics. Using these formulas, we derive Riemannian optimization frameworks on quotients of Stiefel manifolds, including flag manifolds, and a new family of complete quotient metrics on the manifold of positive-semidefinite matrices of fixed rank, considered as a quotient of a product of Stiefel and positive-definite matrix manifold with affine-invariant metrics. The method is procedural, and in many instances, the Riemannian gradient and Hessian formulas could be derived by symbolic calculus. The method extends the list of potential metrics that could be used in manifold optimization and machine learning.