Periodic Regularized Q-Learning
This work addresses a fundamental limitation in reinforcement learning for researchers and practitioners by ensuring convergence under function approximation, though it appears incremental as it builds on existing regularization techniques.
The authors tackled the problem of Q-learning's lack of convergence guarantees under linear function approximation by proposing periodic regularized Q-learning (PRQ), which introduces regularization at the projection operator level to ensure stable convergence, and they provided rigorous theoretical analysis proving finite-time convergence guarantees.
In reinforcement learning (RL), Q-learning is a fundamental algorithm whose convergence is guaranteed in the tabular setting. However, this convergence guarantee does not hold under linear function approximation. To overcome this limitation, a significant line of research has introduced regularization techniques to ensure stable convergence under function approximation. In this work, we propose a new algorithm, periodic regularized Q-learning (PRQ). We first introduce regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm. By appropriately regularizing the projection operator, the resulting projected value iteration becomes a contraction. By extending this regularized projection into the stochastic setting, we establish the PRQ algorithm and provide a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.