Hierarchical Reinforcement Learning for Deep Goal Reasoning: An Expressiveness Analysis
This work addresses expressiveness limitations in hierarchical reinforcement learning frameworks, which is an incremental improvement for AI researchers focusing on goal reasoning and hierarchical control.
The paper identifies tasks that cannot be solved by Hierarchical DQN (h-DQN), a hierarchical reinforcement learning framework, and proposes a more expressive Recurrent Hierarchical Framework (RHF) using recurrent neural networks at the meta level, with experiments showing RHF outperforms HF baselines.
Hierarchical DQN (h-DQN) is a two-level architecture of feedforward neural networks where the meta level selects goals and the lower level takes actions to achieve the goals. We show tasks that cannot be solved by h-DQN, exemplifying the limitation of this type of hierarchical framework (HF). We describe the recurrent hierarchical framework (RHF), generalizing architectures that use a recurrent neural network at the meta level. We analyze the expressiveness of HF and RHF using context-sensitive grammars. We show that RHF is more expressive than HF. We perform experiments comparing an implementation of RHF with two HF baselines; the results corroborate our theoretical findings.