Double Q($σ$) and Q($σ, λ$): Unifying Reinforcement Learning Control Algorithms
This work provides incremental improvements for researchers in reinforcement learning by unifying and extending existing temporal-difference algorithms.
The paper tackled the problem of unifying reinforcement learning control algorithms by extending Q(σ) to Q(σ, λ) with eligibility traces and introducing Double Q(σ) for double learning, with experiments showing Q(σ, λ) can outperform classical methods like Sarsa(λ), Q(λ), and Q(σ).
Temporal-difference (TD) learning is an important field in reinforcement learning. Sarsa and Q-Learning are among the most used TD algorithms. The Q($σ$) algorithm (Sutton and Barto (2017)) unifies both. This paper extends the Q($σ$) algorithm to an online multi-step algorithm Q($σ, λ$) using eligibility traces and introduces Double Q($σ$) as the extension of Q($σ$) to double learning. Experiments suggest that the new Q($σ, λ$) algorithm can outperform the classical TD control methods Sarsa($λ$), Q($λ$) and Q($σ$).