A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithms
This work addresses the theoretical analysis problem for reinforcement learning researchers, offering a novel framework to understand convergence in Q-learning algorithms, which is incremental as it builds on existing switching system theories.
The paper tackles the analysis of Q-learning algorithms by introducing a unified framework using switching system perspectives and ODE-based stochastic approximation, showing that nonlinear ODE models can be formulated as switched linear systems and analyzing their asymptotic stability, providing the first ODE analysis for asymptotic convergence of various Q-learning algorithms including asynchronous and averaging versions, and extending it to linear function approximation with a new sufficient condition for convergence.
In this paper, we introduce a unified framework for analyzing a large family of Q-learning algorithms, based on switching system perspectives and ODE-based stochastic approximation. We show that the nonlinear ODE models associated with these Q-learning algorithms can be formulated as switched linear systems, and analyze their asymptotic stability by leveraging existing switching system theories. Our approach provides the first O.D.E. analysis of the asymptotic convergence of various Q-learning algorithms, including asynchronous Q-learning and averaging Q-learning. We also extend the approach to analyze Q-learning with linear function approximation and derive a new sufficient condition for its convergence.