Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and $λ$-Effectiveness
This work provides incremental theoretical improvements for machine learning practitioners using Kaczmarz algorithms, offering regret bounds that help optimize algorithm performance in noisy and infinite-dimensional settings.
The paper tackles the problem of quantifying performance deviation in projection-based linear regression algorithms, specifically Kaczmarz methods, by establishing a priori regret bounds with explicit dependence on the relaxation parameter λ, and applies these bounds to noisy data scenarios and infinite-dimensional Hilbert spaces.
We present a variety of projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance. We study the role of the relaxation parameter in generalized Kaczmarz algorithms and establish a priori regret bounds with explicit $λ$-dependence to quantify how much an algorithm's performance deviates from its optimal performance. A detailed analysis of relaxation parameter is also provided. Applications include: explicit regret bounds for the framework of Kaczmarz algorithm models, non-orthogonal Fourier expansions, and the use of regret estimates in modern machine learning models, including for noisy data, i.e., regret bounds for the noisy Kaczmarz algorithms. Motivated by machine-learning practice, our wider framework treats bounded operators (on infinite-dimensional Hilbert spaces), with updates realized as (block) Kaczmarz algorithms, leading to new and versatile results.