Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
This addresses safety constraints in offline RL for real-world applications like navigation, but it is incremental as it extends existing reachability methods to cumulative costs.
The paper tackles the problem of balancing reward maximization with cumulative safety constraints in offline reinforcement learning by introducing a safety-conditioned reachability set, and it demonstrates that their method matches or outperforms state-of-the-art baselines while maintaining safety in benchmarks and a real-world maritime navigation task.
Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, that can lead to unstable min/max, adversarial optimization. A promising alternative is safety reachability analysis, which precomputes a forward-invariant safe state, action set, ensuring that an agent starting inside this set remains safe indefinitely. Yet, most reachability based methods address only hard safety constraints, and little work extends reachability to cumulative cost constraints. To address this, first, we define a safetyconditioned reachability set that decouples reward maximization from cumulative safety cost constraints. Second, we show how this set enforces safety constraints without unstable min/max or Lagrangian optimization, yielding a novel offline safe RL algorithm that learns a safe policy from a fixed dataset without environment interaction. Finally, experiments on standard offline safe RL benchmarks, and a real world maritime navigation task demonstrate that our method matches or outperforms state of the art baselines while maintaining safety.