Stable and Efficient Policy Evaluation
This addresses a long-standing bottleneck in reinforcement learning for researchers and practitioners, though it appears incremental as it builds on existing TD methods.
The paper tackled the dual problems of off-policy instability and on-policy inefficiency in reinforcement learning policy evaluation, introducing novel algorithms that achieve both stability and efficiency using oblique projection, with empirical validation across domains.
Policy evaluation algorithms are essential to reinforcement learning due to their ability to predict the performance of a policy. However, there are two long-standing issues lying in this prediction problem that need to be tackled: off-policy stability and on-policy efficiency. The conventional temporal difference (TD) algorithm is known to perform very well in the on-policy setting, yet is not off-policy stable. On the other hand, the gradient TD and emphatic TD algorithms are off-policy stable, but are not on-policy efficient. This paper introduces novel algorithms that are both off-policy stable and on-policy efficient by using the oblique projection method. The empirical experimental results on various domains validate the effectiveness of the proposed approach.