Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
This work addresses a key bottleneck in hierarchical reinforcement learning for continuous control tasks, offering incremental improvements over prior methods.
The paper tackles the challenge of generating effective subgoals in hierarchical reinforcement learning by proposing a method that uses a conditional diffusion model with a Gaussian Process prior to capture complex subgoal distributions and account for uncertainty, resulting in improved sample efficiency and performance on continuous control benchmarks.
Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks.