Polynomial convergence of iterations of certain random operators in Hilbert space
This work addresses theoretical convergence guarantees for iterative methods in machine learning, but it appears incremental as it builds on prior studies of SGD in noiseless regression.
The paper tackles the convergence of random iterative sequences of operators in infinite-dimensional Hilbert spaces, inspired by noiseless SGD, by identifying broader conditions for polynomial convergence rates in various norms and characterizing the role of randomness in determining multiplicative constants, while also proving almost sure convergence.
We study the convergence of a random iterative sequence of a family of operators on infinite dimensional Hilbert spaces, inspired by the Stochastic Gradient Descent (SGD) algorithm in the case of the noiseless regression, as studied in [1]. We identify conditions that are strictly broader than previously known for polynomial convergence rate in various norms, and characterize the roles the randomness plays in determining the best multiplicative constants. Additionally, we prove almost sure convergence of the sequence.