Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning
This addresses the challenge of versatile safe RL for real-world dynamic applications, representing a novel method for a known bottleneck.
The paper tackles the problem of training safe reinforcement learning agents that can adapt to varying safety constraints during deployment without retraining, and demonstrates that their CCPO framework outperforms baselines in safety and task performance while enabling zero-shot adaptation.
Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored and challenging area. In this work, we formulate the versatile safe RL problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines in terms of safety and task performance while preserving zero-shot adaptation capabilities to different constraint thresholds data-efficiently. This makes our approach suitable for real-world dynamic applications.