Hierarchical Orchestra of Policies
This addresses the problem of catastrophic forgetting for lifelong reinforcement learning agents, offering an incremental improvement by eliminating the need for task labeling in ambiguous environments.
The paper tackles catastrophic forgetting in continual reinforcement learning by introducing the Hierarchical Orchestra of Policies (HOP), which dynamically forms a policy hierarchy based on observation similarity, and it significantly outperforms baselines in knowledge retention across tasks while performing comparably to state-of-the-art methods that require task labeling.
Continual reinforcement learning poses a major challenge due to the tendency of agents to experience catastrophic forgetting when learning sequential tasks. In this paper, we introduce a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning. HOP dynamically forms a hierarchy of policies based on a similarity metric between the current observations and previously encountered observations in successful tasks. Unlike other state-of-the-art methods, HOP does not require task labelling, allowing for robust adaptation in environments where boundaries between tasks are ambiguous. Our experiments, conducted across multiple tasks in a procedurally generated suite of environments, demonstrate that HOP significantly outperforms baseline methods in retaining knowledge across tasks and performs comparably to state-of-the-art transfer methods that require task labelling. Moreover, HOP achieves this without compromising performance when tasks remain constant, highlighting its versatility.