Lyapunov-Certified Direct Switching Theory for Q-Learning
Provides a new theoretical framework for understanding Q-learning convergence, offering tighter bounds than standard row-sum analysis, which is relevant for reinforcement learning theorists.
The paper analyzes constant-stepsize Q-learning by modeling it as a direct stochastic switching system, showing that the Bellman maximization error can be exactly represented by a stochastic policy. This yields a finite-time final-iterate bound via a Lyapunov function based on the joint spectral radius, with a computable quadratic-certificate version.
Q-learning is one of the most fundamental algorithms in reinforcement learning. We analyze constant-stepsize Q-learning through a direct stochastic switching system representation. The key observation is that the Bellman maximization error can be represented exactly by a stochastic policy. Therefore, the Q-learning error admits a switched linear conditional-mean recursion with martingale-difference noise. The intrinsic drift rate is the joint spectral radius (JSR) of the direct switching family, which can be strictly smaller than the standard row-sum rate. Using this representation, we derive a finite-time final-iterate bound via a JSR-induced Lyapunov function and then give a computable quadratic-certificate version.