Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
This addresses a key bottleneck in real-life applications like healthcare and robotics by enabling off-policy estimation without knowledge of data collection, though it is incremental as it builds on prior work to remove specific constraints.
The paper tackles the problem of off-policy estimation in infinite-horizon reinforcement learning, where existing methods require data from a known behavior policy's stationary distribution, and proposes a novel estimator that eliminates this limitation, achieving effectiveness as verified in experiments.
Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-fidelity simulators may not be available and on-policy evaluation is expensive or impossible. Recently, \cite{liu18breaking} proposed an approach that avoids the \emph{curse of horizon} suffered by typical importance-sampling-based methods. While showing promising results, this approach is limited in practice as it requires data be drawn from the \emph{stationary distribution} of a \emph{known} behavior policy. In this work, we propose a novel approach that eliminates such limitations. In particular, we formulate the problem as solving for the fixed point of a certain operator. Using tools from Reproducing Kernel Hilbert Spaces (RKHSs), we develop a new estimator that computes importance ratios of stationary distributions, without knowledge of how the off-policy data are collected. We analyze its asymptotic consistency and finite-sample generalization. Experiments on benchmarks verify the effectiveness of our approach.