Heavy-Ball Q-Learning with Residual Weighting Correction
For RL practitioners, this provides a theoretically grounded acceleration technique for Q-learning, though the improvement is conditional and the analysis is incremental.
This paper proposes a corrected heavy-ball Q-learning method for RL and proves its convergence, showing conditions under which it converges faster than standard Q-learning. The method is also extended to linear function approximation with analogous guarantees.
This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence. It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning. The same construction is then extended to Q-learning with linear function approximation, where analogous convergence and acceleration statements are derived. The analysis is based on a switched linear system (SLS) representation of Q-learning algorithms and on the joint spectral radius (JSR) of the associated switching families. This SLS viewpoint is not commonly used in standard analyses of Q-learning, and it provides a complementary framework and new insight into how heavy-ball momentum can accelerate Q-learning.