Distributional Offline Policy Evaluation with Predictive Error Guarantees
This addresses distributional offline policy evaluation for reinforcement learning, enabling more accurate uncertainty quantification in offline settings, though it is incremental as it builds on existing probabilistic generative models.
The paper tackles the problem of estimating the return distribution of a policy using offline data not generated from that policy, proposing Fitted Likelihood Estimation (FLE) which learns distributions close to the ground truth under total variation and Wasserstein distances, with experimental validation using Gaussian mixture and diffusion models for multi-dimensional rewards.
We study the problem of estimating the distribution of the return of a policy using an offline dataset that is not generated from the policy, i.e., distributional offline policy evaluation (OPE). We propose an algorithm called Fitted Likelihood Estimation (FLE), which conducts a sequence of Maximum Likelihood Estimation (MLE) and has the flexibility of integrating any state-of-the-art probabilistic generative models as long as it can be trained via MLE. FLE can be used for both finite-horizon and infinite-horizon discounted settings where rewards can be multi-dimensional vectors. Our theoretical results show that for both finite-horizon and infinite-horizon discounted settings, FLE can learn distributions that are close to the ground truth under total variation distance and Wasserstein distance, respectively. Our theoretical results hold under the conditions that the offline data covers the test policy's traces and that the supervised learning MLE procedures succeed. Experimentally, we demonstrate the performance of FLE with two generative models, Gaussian mixture models and diffusion models. For the multi-dimensional reward setting, FLE with diffusion models is capable of estimating the complicated distribution of the return of a test policy.