Numerical Instabilities in the Kaczmarz Method and Stabilization by Iterative Refinement
For researchers using randomized Kaczmarz methods for large-scale linear systems, this work addresses a critical numerical stability issue and offers a practical fix.
The paper identifies that classical and accelerated randomized Kaczmarz methods are not forward stable, and proposes iterative refinement to stabilize them, achieving high-accuracy solutions even for ill-conditioned systems.
The randomized Kaczmarz method and its accelerated variants are a powerful class of iterative methods for solving large-scale linear systems, offering guaranteed convergence with low per-iteration cost. However, their numerical stability remains poorly understood. In this work, we investigate the stability properties of both classical and accelerated randomized Kaczmarz methods, with an emphasis on how error propagates across iterations and interacts with acceleration. We show that both classical and accelerated randomized Kaczmarz fail to be forward stable. To address this issue, we propose the integration of iterative refinement into randomized Kaczmarz frameworks. We demonstrate that refinement can effectively control error accumulation and recover high-accuracy solutions, even when the system is ill-conditioned. Numerical experiments corroborate our theoretical findings and illustrate the practical benefits of combining refinement with both classical and accelerated Kaczmarz methods.