Average-Reward Off-Policy Policy Evaluation with Function Approximation
This work provides convergent off-policy policy evaluation algorithms for average-reward MDPs, which is a significant step for researchers and practitioners working with reinforcement learning in continuous or long-running tasks where the average reward is a more suitable metric.
This paper addresses off-policy policy evaluation with function approximation in average-reward Markov Decision Processes, aiming to estimate both the reward rate and the differential value function. The authors propose two novel algorithms that are the first convergent off-policy linear function approximation algorithms for estimating the differential value function, and the first for estimating the reward rate without requiring density ratio estimation. Empirical results demonstrate their advantage over a density-ratio-based approach in simple and robot simulation tasks.
We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function. For this problem, bootstrapping is necessary and, along with off-policy learning and FA, results in the deadly triad (Sutton & Barto, 2018). To address the deadly triad, we propose two novel algorithms, reproducing the celebrated success of Gradient TD algorithms in the average-reward setting. In terms of estimating the differential value function, the algorithms are the first convergent off-policy linear function approximation algorithms. In terms of estimating the reward rate, the algorithms are the first convergent off-policy linear function approximation algorithms that do not require estimating the density ratio. We demonstrate empirically the advantage of the proposed algorithms, as well as their nonlinear variants, over a competitive density-ratio-based approach, in a simple domain as well as challenging robot simulation tasks.