Iterative methods for solving factorized linear systems
This work provides a faster iterative solver for large-scale linear systems with low-rank factorized matrices, benefiting applications in data analysis and machine learning where storage is limited.
The paper proposes a variant of the randomized Kaczmarz method for solving linear systems where the matrix is stored in factorized form (X = UV), avoiding explicit computation of X. The method achieves exponential convergence and significant acceleration in experiments.
Stochastic iterative algorithms such as the Kaczmarz and Gauss-Seidel methods have gained recent attention because of their speed, simplicity, and the ability to approximately solve large-scale linear systems of equations without needing to access the entire matrix. In this work, we consider the setting where we wish to solve a linear system in a large matrix X that is stored in a factorized form, X = UV; this setting either arises naturally in many applications or may be imposed when working with large low-rank datasets for reasons of space required for storage. We propose a variant of the randomized Kaczmarz method for such systems that takes advantage of the factored form, and avoids computing X. We prove an exponential convergence rate and supplement our theoretical guarantees with experimental evidence demonstrating that the factored variant yields significant acceleration in convergence.