Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise
This work addresses foundational challenges in reinforcement learning theory by providing rigorous convergence guarantees for algorithms under realistic Markovian noise conditions.
The paper tackles the problem of establishing almost sure convergence rates and concentration bounds for stochastic approximation algorithms with Markovian noise, achieving the first such results for general contractive algorithms and applying them to Q-learning and off-policy TD learning.
This paper establishes the first almost sure convergence rate and the first maximal concentration bound with exponential tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in $L^p$. Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. As applications, we provide the first almost sure convergence rate for $Q$-learning with Markovian samples without count-based learning rates. We also provide the first concentration bound for off-policy temporal difference learning with Markovian samples.