Learning Multi-Objective Curricula for Robotic Policy Learning
This addresses the challenge of improving sample efficiency and performance in robotic policy learning by integrating complementary curriculum paradigms, though it appears incremental as it builds on existing ACL methods.
The paper tackles the problem of automatic curriculum learning (ACL) in deep reinforcement learning by proposing a unified framework that combines multiple curriculum paradigms (e.g., subgoal generation, reward shaping) into coherent multi-objective curricula, using a multi-task hyper-net to coordinate potentially conflicting modules. The method demonstrates superiority over state-of-the-art ACL methods on robotic manipulation tasks, with improvements in sample efficiency and final performance.
Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL). They are designed to control how a DRL agent collects data, which is inspired by how humans gradually adapt their learning processes to their capabilities. For example, ACL can be used for subgoal generation, reward shaping, environment generation, or initial state generation. However, prior work only considers curriculum learning following one of the aforementioned predefined paradigms. It is unclear which of these paradigms are complementary, and how the combination of them can be learned from interactions with the environment. Therefore, in this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula that are generated by a set of parametric curriculum modules. Each curriculum module is instantiated as a neural network and is responsible for generating a particular curriculum. In order to coordinate those potentially conflicting modules in unified parameter space, we propose a multi-task hyper-net learning framework that uses a single hyper-net to parameterize all those curriculum modules. In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum, which may otherwise be difficult to design manually. We evaluate our method on a series of robotic manipulation tasks and demonstrate its superiority over other state-of-the-art ACL methods in terms of sample efficiency and final performance.