Discerning Temporal Difference Learning
This work addresses a specific bottleneck in reinforcement learning for researchers and practitioners, offering an incremental improvement over existing TD methods.
The paper tackles the problem of inefficient value estimation in temporal difference learning by proposing discerning TD learning (DTD), which uses flexible emphasis functions to allocate efforts across states, resulting in improved value estimation and faster learning in diverse scenarios.
Temporal difference learning (TD) is a foundational concept in reinforcement learning (RL), aimed at efficiently assessing a policy's value function. TD($λ$), a potent variant, incorporates a memory trace to distribute the prediction error into the historical context. However, this approach often neglects the significance of historical states and the relative importance of propagating the TD error, influenced by challenges such as visitation imbalance or outcome noise. To address this, we propose a novel TD algorithm named discerning TD learning (DTD), which allows flexible emphasis functions$-$predetermined or adapted during training$-$to allocate efforts effectively across states. We establish the convergence properties of our method within a specific class of emphasis functions and showcase its promising potential for adaptation to deep RL contexts. Empirical results underscore that employing a judicious emphasis function not only improves value estimation but also expedites learning across diverse scenarios.