Imagine to Ensure Safety in Hierarchical Reinforcement Learning
For RL practitioners dealing with long-horizon safe exploration, this method addresses compounding estimation errors and restricted exploration, offering strong empirical safety guarantees.
This work proposes a hierarchical safe RL method combining a world model with high- and low-level policies to ensure safety in long-horizon tasks. It significantly outperforms baselines in success rate and constraint satisfaction, consistently meeting safety budgets across seeds.
This work investigates the safe exploration problem in reinforcement learning, where an agent must maximize cumulative performance while simultaneously satisfying safety constraints. This challenge becomes even more pronounced in long-horizon tasks, where existing safe methods face fundamental limitations due to compounding estimation errors and restricted exploration capabilities. To address this problem, we propose a method that combines a learnable world model with two complementary policies a high-level policy and a low-level policy to promote safety at both hierarchical levels. The high-level policy generates intermediate subgoals that bias exploration toward safe regions, while the low-level policy uses imagined rollouts in the learned world model to reduce unsafe behaviors when reaching these subgoals. The proposed method was evaluated on challenging long-horizon navigation and manipulation tasks with high-dimensional action spaces, where it significantly outperforms existing Safe RL baselines in both success rate and strong empirical constraint satisfaction, consistently meeting the prescribed safety budget across seeds, while prior approaches fail to effectively solve these complex long-horizon scenarios.