A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning
This addresses the problem of learning safe and efficient policies from fixed datasets for applications like robotics or healthcare, representing an incremental improvement by relaxing assumptions.
The paper tackles offline constrained reinforcement learning by proposing the Primal-Dual-Critic Algorithm (PDCA), which learns a policy to maximize reward under cost constraints using a dataset, and shows it finds a near saddle point for near-optimal solutions with sample efficiency under weaker assumptions than prior work.
Offline constrained reinforcement learning (RL) aims to learn a policy that maximizes the expected cumulative reward subject to constraints on expected cumulative cost using an existing dataset. In this paper, we propose Primal-Dual-Critic Algorithm (PDCA), a novel algorithm for offline constrained RL with general function approximation. PDCA runs a primal-dual algorithm on the Lagrangian function estimated by critics. The primal player employs a no-regret policy optimization oracle to maximize the Lagrangian estimate and the dual player acts greedily to minimize the Lagrangian estimate. We show that PDCA can successfully find a near saddle point of the Lagrangian, which is nearly optimal for the constrained RL problem. Unlike previous work that requires concentrability and a strong Bellman completeness assumption, PDCA only requires concentrability and realizability assumptions for sample-efficient learning.