Offline Reinforcement Learning with Additional Covering Distributions
This work addresses the problem of learning optimal policies from logged datasets for researchers and practitioners in reinforcement learning, offering a method that relaxes strong assumptions like full coverage or realizability, though it is incremental as it builds on existing importance sampling techniques.
The paper tackles the challenge of offline reinforcement learning with function approximation by providing finite-sample guarantees for a marginalized importance sampling algorithm, showing that sample-efficient learning is possible with only partial coverage and weak realizable function classes when given a covering distribution as side information.
We study learning optimal policies from a logged dataset, i.e., offline RL, with function approximation. Despite the efforts devoted, existing algorithms with theoretic finite-sample guarantees typically assume exploratory data coverage or strong realizable function classes, which is hard to be satisfied in reality. While there are recent works that successfully tackle these strong assumptions, they either require the gap assumptions that only could be satisfied by part of MDPs or use the behavior regularization that makes the optimality of learned policy even intractable. To solve this challenge, we provide finite-sample guarantees for a simple algorithm based on marginalized importance sampling (MIS), showing that sample-efficient offline RL for general MDPs is possible with only a partial coverage dataset and weak realizable function classes given additional side information of a covering distribution. Furthermore, we demonstrate that the covering distribution trades off prior knowledge of the optimal trajectories against the coverage requirement of the dataset, revealing the effect of this inductive bias in the learning processes.