Hierarchical Reinforcement Learning with Deep Nested Agents
This work addresses performance issues in hierarchical reinforcement learning for complex domains like Minecraft, representing an incremental improvement over existing methods.
The paper tackles the inefficiency of deep hierarchical reinforcement learning in complex environments by introducing the Deep Nested Agent framework, which propagates information from the main agent to a nested agent to improve performance, demonstrating effectiveness in three Minecraft scenarios with comparisons to non-hierarchical and hierarchical frameworks.
Deep hierarchical reinforcement learning has gained a lot of attention in recent years due to its ability to produce state-of-the-art results in challenging environments where non-hierarchical frameworks fail to learn useful policies. However, as problem domains become more complex, deep hierarchical reinforcement learning can become inefficient, leading to longer convergence times and poor performance. We introduce the Deep Nested Agent framework, which is a variant of deep hierarchical reinforcement learning where information from the main agent is propagated to the low level $nested$ agent by incorporating this information into the nested agent's state. We demonstrate the effectiveness and performance of the Deep Nested Agent framework by applying it to three scenarios in Minecraft with comparisons to a deep non-hierarchical single agent framework, as well as, a deep hierarchical framework.