Backstepping Temporal Difference Learning
This work addresses a fundamental stability problem in reinforcement learning for practical applications, offering an incremental improvement over existing off-policy TD methods.
The paper tackled the divergence issue of off-policy temporal-difference learning with linear function approximation by proposing a new convergent algorithm based on the backstepping technique from nonlinear control theory, and experimentally verified its convergence in environments where standard TD-learning is unstable.
Off-policy learning ability is an important feature of reinforcement learning (RL) for practical applications. However, even one of the most elementary RL algorithms, temporal-difference (TD) learning, is known to suffer form divergence issue when the off-policy scheme is used together with linear function approximation. To overcome the divergent behavior, several off-policy TD-learning algorithms, including gradient-TD learning (GTD), and TD-learning with correction (TDC), have been developed until now. In this work, we provide a unified view of such algorithms from a purely control-theoretic perspective, and propose a new convergent algorithm. Our method relies on the backstepping technique, which is widely used in nonlinear control theory. Finally, convergence of the proposed algorithm is experimentally verified in environments where the standard TD-learning is known to be unstable.