On Time-Indexing as Inductive Bias in Deep RL for Sequential Manipulation Tasks
This addresses sample efficiency issues in robotic manipulation for researchers, but it is incremental as it builds on existing policy-learning methods with a simple structural modification.
The paper tackles the problem of low sample efficiency and poor performance in deep reinforcement learning for multimodal sequential manipulation tasks by proposing a policy architecture that sequentially executes different action heads for fixed durations. The result is improved performance on Metaworld tasks compared to standard methods.
While solving complex manipulation tasks, manipulation policies often need to learn a set of diverse skills to accomplish these tasks. The set of skills is often quite multimodal - each one may have a quite distinct distribution of actions and states. Standard deep policy-learning algorithms often model policies as deep neural networks with a single output head (deterministic or stochastic). This structure requires the network to learn to switch between modes internally, which can lead to lower sample efficiency and poor performance. In this paper we explore a simple structure which is conducive to skill learning required for so many of the manipulation tasks. Specifically, we propose a policy architecture that sequentially executes different action heads for fixed durations, enabling the learning of primitive skills such as reaching and grasping. Our empirical evaluation on the Metaworld tasks reveals that this simple structure outperforms standard policy learning methods, highlighting its potential for improved skill acquisition.