On the benefits of pixel-based hierarchical policies for task generalization
This addresses the challenge of building more adaptable reinforcement learning architectures for multi-task robotic control, though it is incremental as it builds on existing hierarchical policy concepts.
The paper tackles the problem of task generalization in reinforcement learning by analyzing hierarchical policies in pixel-based robotic control, showing they improve training task performance, enhance generalization to similar tasks, and reduce fine-tuning complexity for novel tasks.
Reinforcement learning practitioners often avoid hierarchical policies, especially in image-based observation spaces. Typically, the single-task performance improvement over flat-policy counterparts does not justify the additional complexity associated with implementing a hierarchy. However, by introducing multiple decision-making levels, hierarchical policies can compose lower-level policies to more effectively generalize between tasks, highlighting the need for multi-task evaluations. We analyze the benefits of hierarchy through simulated multi-task robotic control experiments from pixels. Our results show that hierarchical policies trained with task conditioning can (1) increase performance on training tasks, (2) lead to improved reward and state-space generalizations in similar tasks, and (3) decrease the complexity of fine tuning required to solve novel tasks. Thus, we believe that hierarchical policies should be considered when building reinforcement learning architectures capable of generalizing between tasks.