HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learning
This addresses exploration and long-horizon planning problems for robotics and simulation tasks, representing an incremental improvement by combining existing hierarchical and exploration methods.
The paper tackles the challenge of sparse rewards and long time horizons in reinforcement learning by proposing HAC Explore (HACx), which integrates Random Network Distillation (RND) into Hierarchical Actor-Critic (HAC), outperforming existing methods and solving a sparse reward, continuous-control task requiring over 1,000 actions.
Sparse rewards and long time horizons remain challenging for reinforcement learning algorithms. Exploration bonuses can help in sparse reward settings by encouraging agents to explore the state space, while hierarchical approaches can assist with long-horizon tasks by decomposing lengthy tasks into shorter subtasks. We propose HAC Explore (HACx), a new method that combines these approaches by integrating the exploration bonus method Random Network Distillation (RND) into the hierarchical approach Hierarchical Actor-Critic (HAC). HACx outperforms either component method on its own, as well as an existing approach to combining hierarchy and exploration, in a set of difficult simulated robotics tasks. HACx is the first RL method to solve a sparse reward, continuous-control task that requires over 1,000 actions.