Investigating practical linear temporal difference learning
This work addresses a specific algorithmic gap in off-policy reinforcement learning, offering incremental improvements for researchers and practitioners in areas like learning from demonstration.
The paper derived two new hybrid temporal difference (TD) policy-evaluation algorithms to fill a gap in off-policy reinforcement learning and empirically compared them to provide practical recommendations for different situations.
Off-policy reinforcement learning has many applications including: learning from demonstration, learning multiple goal seeking policies in parallel, and representing predictive knowledge. Recently there has been an proliferation of new policy-evaluation algorithms that fill a longstanding algorithmic void in reinforcement learning: combining robustness to off-policy sampling, function approximation, linear complexity, and temporal difference (TD) updates. This paper contains two main contributions. First, we derive two new hybrid TD policy-evaluation algorithms, which fill a gap in this collection of algorithms. Second, we perform an empirical comparison to elicit which of these new linear TD methods should be preferred in different situations, and make concrete suggestions about practical use.