Asymptotic regularity of a generalised stochastic Halpern scheme
This work provides incremental theoretical analysis for stochastic optimization methods, potentially benefiting researchers in optimization and reinforcement learning.
The paper tackles the problem of analyzing asymptotic regularity for a generalized stochastic Halpern iteration, incorporating stochasticity and a second mapping, and obtains linear rates for specific cases matching or improving known rates, and quadratic rates in inner product spaces.
We provide abstract, general and highly uniform rates of asymptotic regularity for a generalized stochastic Halpern-style iteration, which incorporates a second mapping in the style of a Krasnoselskii-Mann iteration. This iteration is general in two ways: First, it incorporates stochasticity in a completely abstract way rather than fixing a sampling method; secondly, it includes as special cases stochastic versions of various schemes from the optimization literature, including Halpern's iteration as well as a Krasnoselskii-Mann iteration with Tikhonov regularization terms in the sense of Boţ, Csetnek and Meier. For these specific cases, we in particular obtain linear rates of asymptotic regularity, matching (or improving) the currently best known rates for these iterations in stochastic optimization, and quadratic rates of asymptotic regularity are obtained in the context of inner product spaces for the general iteration. At the end, we briefly sketch how the schemes presented here can be instantiated in the context of reinforcement learning to yield novel methods for Q-learning.