Hierarchical Skills for Efficient Exploration
This addresses the need for prior knowledge in skill design for efficient exploration in reinforcement learning, particularly for bipedal robots, though it is incremental as it builds on existing hierarchical methods.
The paper tackles the challenge of balancing generality and specificity in pre-trained low-level skills for reinforcement learning by introducing a hierarchical skill learning framework that acquires skills unsupervised and a three-layered algorithm for task-specific trade-offs, achieving better results than state-of-the-art methods.
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensitivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest. In this work, we analyze this trade-off for low-level policy pre-training with a new benchmark suite of diverse, sparse-reward tasks for bipedal robots. We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner. For utilization on downstream tasks, we present a three-layered hierarchical learning algorithm to automatically trade off between general and specific skills as required by the respective task. In our experiments, we show that our approach performs this trade-off effectively and achieves better results than current state-of-the-art methods for end- to-end hierarchical reinforcement learning and unsupervised skill discovery. Code and videos are available at https://facebookresearch.github.io/hsd3 .