Temporal-adaptive Hierarchical Reinforcement Learning
This work addresses a specific bottleneck in hierarchical reinforcement learning for researchers and practitioners, offering an incremental improvement over fixed-time skip strategies.
The paper tackled the problem of determining optimal decision frequencies for high-level policies in hierarchical reinforcement learning by proposing a temporal-adaptive structure (TEMPLE) that uses a temporal gate to adaptively control frequency, resulting in improved performance across environments like 2-D rooms, Mujoco tasks, and Atari games.
Hierarchical reinforcement learning (HRL) helps address large-scale and sparse reward issues in reinforcement learning. In HRL, the policy model has an inner representation structured in levels. With this structure, the reinforcement learning task is expected to be decomposed into corresponding levels with sub-tasks, and thus the learning can be more efficient. In HRL, although it is intuitive that a high-level policy only needs to make macro decisions in a low frequency, the exact frequency is hard to be simply determined. Previous HRL approaches often employed a fixed-time skip strategy or learn a terminal condition without taking account of the context, which, however, not only requires manual adjustments but also sacrifices some decision granularity. In this paper, we propose the \emph{temporal-adaptive hierarchical policy learning} (TEMPLE) structure, which uses a temporal gate to adaptively control the high-level policy decision frequency. We train the TEMPLE structure with PPO and test its performance in a range of environments including 2-D rooms, Mujoco tasks, and Atari games. The results show that the TEMPLE structure can lead to improved performance in these environments with a sequential adaptive high-level control.