Stochastic Halpern Iteration with Variance Reduction for Stochastic Monotone Inclusions
This work addresses computational efficiency challenges in stochastic optimization for machine learning practitioners, offering incremental improvements over existing methods.
The paper tackles stochastic monotone inclusion problems, such as those in robust regression and adversarial learning, by proposing novel variants of stochastic Halpern iteration with recursive variance reduction, achieving an improved complexity of O(1/ε^3) stochastic operator evaluations compared to the state-of-the-art O(1/ε^4) for cocoercive and Lipschitz-monotone setups, and further reducing it to O(log(1/ε)/ε^2) under sharpness or strong monotonicity assumptions.
We study stochastic monotone inclusion problems, which widely appear in machine learning applications, including robust regression and adversarial learning. We propose novel variants of stochastic Halpern iteration with recursive variance reduction. In the cocoercive -- and more generally Lipschitz-monotone -- setup, our algorithm attains $ε$ norm of the operator with $\mathcal{O}(\frac{1}{ε^3})$ stochastic operator evaluations, which significantly improves over state of the art $\mathcal{O}(\frac{1}{ε^4})$ stochastic operator evaluations required for existing monotone inclusion solvers applied to the same problem classes. We further show how to couple one of the proposed variants of stochastic Halpern iteration with a scheduled restart scheme to solve stochastic monotone inclusion problems with ${\mathcal{O}}(\frac{\log(1/ε)}{ε^2})$ stochastic operator evaluations under additional sharpness or strong monotonicity assumptions.