Multitask Adaptation by Retrospective Exploration with Learned World Models
This addresses the problem of multitask adaptation in reinforcement learning for researchers and practitioners, offering an incremental improvement over existing MBRL methods.
The paper tackles the problem of sample inefficiency in model-based reinforcement learning (MBRL) when transferring knowledge between tasks, proposing a meta-learned addressing model (RAMa) that selects promising trajectories from task-agnostic storage to accelerate learning. The result shows improved learning speed in domains like DeepMind control suite, Metaworld benchmark, and a photorealistic robotic simulator.
Model-based reinforcement learning (MBRL) allows solving complex tasks in a sample-efficient manner. However, no information is reused between the tasks. In this work, we propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from continuously growing task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage. We show that such retrospective exploration can accelerate the learning process of the MBRL agent by better informing learned dynamics and prompting agent with exploratory trajectories. We test the performance of our approach on several domains from the DeepMind control suite, from Metaworld multitask benchmark, and from our bespoke environment implemented with a robotic NVIDIA Isaac simulator to test the ability of the model to act in a photorealistic, ray-traced environment.