A Principled Path to Fitted Distributional Evaluation
This provides a principled approach for estimating return distributions in reinforcement learning, addressing a gap in distributional off-policy evaluation methods.
The authors tackled the lack of a unified framework for distributional off-policy evaluation in reinforcement learning by developing fitted distributional evaluation (FDE) methods with guiding principles, achieving superior performance in simulations on linear quadratic regulators and Atari games.
In reinforcement learning, distributional off-policy evaluation (OPE) focuses on estimating the return distribution of a target policy using offline data collected under a different policy. This work focuses on extending the widely used fitted Q-evaluation -- developed for expectation-based reinforcement learning -- to the distributional OPE setting. We refer to this extension as fitted distributional evaluation (FDE). While only a few related approaches exist, there remains no unified framework for designing FDE methods. To fill this gap, we present a set of guiding principles for constructing theoretically grounded FDE methods. Building on these principles, we develop several new FDE methods with convergence analysis and provide theoretical justification for existing methods, even in non-tabular environments. Extensive experiments, including simulations on linear quadratic regulators and Atari games, demonstrate the superior performance of the FDE methods.