Riemannian Optimization for Hadamard Products of Low-Rank Matrices
For practitioners in machine learning and data analysis, this work provides an efficient geometric approach to a parameter-efficient matrix model, though the improvements are incremental over existing methods.
The paper tackles the problem of learning Hadamard products of low-rank matrices, which are useful for modeling multiplicative structures but suffer from symmetries under coupled scalings. They propose a Riemannian optimization method with a novel block-diagonal metric and a Gauss-Newton step size, achieving linear scaling per iteration and showing improved performance on real and synthetic datasets.
The elementwise Hadamard product of two low-rank matrices provides a parameter-efficient model for data with multiplicative structure, but its modeling is challenging due to the presence of additional symmetries under coupled row/column scalings between the two factors. In order to leverage the geometry of the space, we formulate the learning of such matrices as optimization on a Riemannian quotient manifold. We propose a novel block-diagonal Riemannian metric derived from the pullback of the Frobenius inner product. The metric is shown to be invariant under the full symmetry group. We develop a Riemannian gradient descent algorithm that uses a tuning-free Gauss--Newton step size and scales linearly in the number of observed entries per iteration. Experiments on real and synthetic datasets illustrate the efficacy of our proposed Riemannian approach.