Offline Constrained Reinforcement Learning under Partial Data Coverage
This addresses the problem of learning safe policies from limited data for researchers and practitioners in reinforcement learning, representing an incremental improvement over existing algorithms.
The paper tackles offline constrained reinforcement learning with general function approximation, proposing an oracle-efficient primal-dual algorithm that achieves O(ε^{-2}) sample complexity under partial data coverage, removing the need for regularization used in prior methods.
We study offline constrained reinforcement learning (RL) with general function approximation. We aim to learn a policy from a pre-collected dataset that maximizes the expected discounted cumulative reward for a primary reward signal while ensuring that expected discounted returns for multiple auxiliary reward signals are above predefined thresholds. Existing algorithms either require fully exploratory data, are computationally inefficient, or depend on an additional auxiliary function classes to obtain an $ε$-optimal policy with sample complexity $O(ε^{-2})$. In this paper, we propose an oracle-efficient primal-dual algorithm based on a linear programming (LP) formulation, achieving $O(ε^{-2})$ sample complexity under partial data coverage. By introducing a realizability assumption, our approach ensures that all saddle points of the Lagrangian are optimal, removing the need for regularization that complicated prior analyses. Through Lagrangian decomposition, our method extracts policies without requiring knowledge of the data-generating distribution, enhancing practical applicability.