A First Empirical Study of Emphatic Temporal Difference Learning
It addresses the problem of stable convergence in reinforcement learning algorithms for researchers, but is incremental as it provides an empirical validation of an existing theoretical algorithm.
This paper presents the first empirical study of Emphatic Temporal-Difference Learning (ETD), comparing it to TD(0) on on-policy and off-policy Mountain Car tasks, finding that ETD achieved better asymptotic error levels without the temporary 'bounce' seen in TD(0), though it was significantly slower in off-policy cases.
In this paper we present the first empirical study of the emphatic temporal-difference learning algorithm (ETD), comparing it with conventional temporal-difference learning, in particular, with linear TD(0), on on-policy and off-policy variations of the Mountain Car problem. The initial motivation for developing ETD was that it has good convergence properties under off-policy training (Sutton, Mahmood and White 2016), but it is also a new algorithm for the on-policy case. In both our on-policy and off-policy experiments, we found that each method converged to a characteristic asymptotic level of error, with ETD better than TD(0). TD(0) achieved a still lower error level temporarily before falling back to its higher asymptote, whereas ETD never showed this kind of "bounce". In the off-policy case (in which TD(0) is not guaranteed to converge), ETD was significantly slower.