Revisiting a Design Choice in Gradient Temporal Difference Learning
This work addresses a foundational challenge in reinforcement learning for researchers and practitioners, offering a simpler and more tunable alternative to existing methods, though it is incremental as it revisits and improves upon a known approach.
The paper tackles the instability problem in off-policy reinforcement learning with function approximation and bootstrapping, known as the deadly triad, by proving that a variant of the previously dismissed $A^\top$TD algorithm, called $A_t^\top$TD, is an effective solution with only one set of parameters and one learning rate, compared to GTD's two sets and two learning rates.
Off-policy learning enables a reinforcement learning (RL) agent to reason counterfactually about policies that are not executed and is one of the most important ideas in RL. It, however, can lead to instability when combined with function approximation and bootstrapping, two arguably indispensable ingredients for large-scale reinforcement learning. This is the notorious deadly triad. The seminal work Sutton et al. (2008) pioneers Gradient Temporal Difference learning (GTD) as the first solution to the deadly triad, which has enjoyed massive success thereafter. During the derivation of GTD, some intermediate algorithm, called $A^\top$TD, was invented but soon deemed inferior. In this paper, we revisit this $A^\top$TD and prove that a variant of $A^\top$TD, called $A_t^\top$TD, is also an effective solution to the deadly triad. Furthermore, this $A_t^\top$TD only needs one set of parameters and one learning rate. By contrast, GTD has two sets of parameters and two learning rates, making it hard to tune in practice. We provide asymptotic analysis for $A^\top_t$TD and finite sample analysis for a variant of $A^\top_t$TD that additionally involves a projection operator. The convergence rate of this variant is on par with the canonical on-policy temporal difference learning.