Randomized Kaczmarz Methods with Beyond-Krylov Convergence
This work addresses a bottleneck in linear system solvers for computational mathematics and engineering, offering an incremental improvement by enhancing Kaczmarz methods to rival Krylov methods in performance.
The paper tackles the problem of slow convergence of randomized Kaczmarz methods compared to Krylov methods by introducing Kaczmarz++, an accelerated algorithm that exploits outlying singular values to achieve fast Krylov-style convergence, provably faster for certain systems and competitive in operations on benchmarks.
Randomized Kaczmarz methods form a family of linear system solvers which converge by repeatedly projecting their iterates onto randomly sampled equations. While effective in some contexts, such as highly over-determined least squares, Kaczmarz methods are traditionally deemed secondary to Krylov subspace methods, since this latter family of solvers can exploit outliers in the input's singular value distribution to attain fast convergence on ill-conditioned systems. In this paper, we introduce Kaczmarz++, an accelerated randomized block Kaczmarz algorithm that exploits outlying singular values in the input to attain a fast Krylov-style convergence. Moreover, we show that Kaczmarz++ captures large outlying singular values provably faster than popular Krylov methods, for both over- and under-determined systems. We also develop an optimized variant for positive semidefinite systems, called CD++, demonstrating empirically that it is competitive in arithmetic operations with both CG and GMRES on a collection of benchmark problems. To attain these results, we introduce several novel algorithmic improvements to the Kaczmarz framework, including adaptive momentum acceleration, Tikhonov-regularized projections, and a memoization scheme for reusing information from previously sampled equation blocks.