Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps
This addresses a theoretical challenge in offline RL for researchers, offering a novel approach to handle weak function approximation, though it is incremental as it builds on existing gap assumptions from other RL areas.
The paper tackles the problem of offline reinforcement learning with insufficient data coverage and only realizability assumptions, by introducing a gap assumption to provide sample-efficiency guarantees for a pessimistic algorithm based on marginalized importance sampling.
We consider a challenging theoretical problem in offline reinforcement learning (RL): obtaining sample-efficiency guarantees with a dataset lacking sufficient coverage, under only realizability-type assumptions for the function approximators. While the existing theory has addressed learning under realizability and under non-exploratory data separately, no work has been able to address both simultaneously (except for a concurrent work which we compare in detail). Under an additional gap assumption, we provide guarantees to a simple pessimistic algorithm based on a version space formed by marginalized importance sampling (MIS), and the guarantee only requires the data to cover the optimal policy and the function classes to realize the optimal value and density-ratio functions. While similar gap assumptions have been used in other areas of RL theory, our work is the first to identify the utility and the novel mechanism of gap assumptions in offline RL with weak function approximation.