Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds
This work addresses a practical bottleneck in Riemannian optimization for researchers and practitioners in scientific computing and machine learning, offering an incremental improvement by automating gradient and Hessian computations.
The paper tackles the challenge of efficiently computing Riemannian gradients and Hessian-vector products for optimization on low-rank matrix and tensor-train manifolds, proposing a method based on automatic differentiation to address cases where analytic formulas are unavailable or impractical.
In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding low-rank approximations is to use Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this paper, we build upon automatic differentiation and propose a method that, given an implementation of the function to be minimized, efficiently computes Riemannian gradients and matrix-by-vector products between an approximate Riemannian Hessian and a given vector.