Decentralized Riemannian natural gradient methods with Kronecker-product approximations
This work addresses large-scale decentralized optimization problems for applications like distributed machine learning, though it appears incremental as it extends existing natural gradient methods to a Riemannian setting with structured approximations.
The paper tackles decentralized optimization on Riemannian manifolds by proposing a decentralized Riemannian natural gradient descent method that approximates the Riemannian Fisher information matrix using Kronecker products to reduce communication costs, achieving convergence with a rate of O(1/K) and demonstrating efficiency in numerical experiments.
With a computationally efficient approximation of the second-order information, natural gradient methods have been successful in solving large-scale structured optimization problems. We study the natural gradient methods for the large-scale decentralized optimization problems on Riemannian manifolds, where the local objective function defined by the local dataset is of a log-probability type. By utilizing the structure of the Riemannian Fisher information matrix (RFIM), we present an efficient decentralized Riemannian natural gradient descent (DRNGD) method. To overcome the communication issue of the high-dimension RFIM, we consider a class of structured problems for which the RFIM can be approximated by a Kronecker product of two low-dimension matrices. By performing the communications over the Kronecker factors, a high-quality approximation of the RFIM can be obtained in a low cost. We prove that DRNGD converges to a stationary point with the best-known rate of $\mathcal{O}(1/K)$. Numerical experiments demonstrate the efficiency of our proposed method compared with the state-of-the-art ones. To the best of our knowledge, this is the first Riemannian second-order method for solving decentralized manifold optimization problems.