Non-Asymptotic Convergence of Stochastic Iterative Algorithms: A Lyapunov Framework
This work provides a unified and self-contained roadmap for the finite-time analysis of stochastic approximation algorithms, particularly for researchers and practitioners in reinforcement learning, by consolidating existing Lyapunov-based techniques.
This paper surveys Lyapunov-based techniques for the finite-time analysis of stochastic iterative algorithms (stochastic approximation algorithms) used to solve fixed-point equations where the operator is accessed through a noisy oracle. It demonstrates how generalized Moreau envelopes can serve as universal Lyapunov functions, providing mean-square convergence guarantees for algorithms like stochastic gradient descent, linear SA, Q-learning, and temporal-difference learning.
We survey Lyapunov-based techniques for the finite-time analysis of stochastic iterative algorithms, also known as stochastic approximation (SA) algorithms, for solving fixed-point equations $\bar{F}(x)=x$, where the operator $\bar{F}(\cdot)$ can only be accessed through a noisy oracle. We first focus on the standard setting in which $\bar{F}(\cdot)$ is contractive with respect to some norm and the noise is i.i.d., and explain how generalized Moreau envelopes serve as universal Lyapunov functions, regardless of the underlying norm. We then show how this framework yields mean-square convergence guarantees and applies to stochastic gradient descent, linear SA, and value-based reinforcement learning algorithms such as Q-learning and temporal-difference learning. Finally, we discuss extensions to Markovian noise, seminorm-contractive operators, dissipative operators, and high-probability bounds, and conclude with open problems. The goal is to present a unified and self-contained roadmap for the finite-time analysis of SA and its applications, especially in reinforcement learning.