Curious Replay for Model-based Adaptation
This addresses the adaptation challenge for model-based RL agents, offering an incremental improvement over existing replay methods.
The paper tackles the problem of slow adaptation in model-based reinforcement learning agents by introducing Curious Replay, a prioritized experience replay method using a curiosity-based signal, which improves performance on the Crafter benchmark with a mean score of 19.4 compared to the previous high of 14.5.
Agents must be able to adapt quickly as an environment changes. We find that existing model-based reinforcement learning agents are unable to do this well, in part because of how they use past experiences to train their world model. Here, we present Curious Replay -- a form of prioritized experience replay tailored to model-based agents through use of a curiosity-based priority signal. Agents using Curious Replay exhibit improved performance in an exploration paradigm inspired by animal behavior and on the Crafter benchmark. DreamerV3 with Curious Replay surpasses state-of-the-art performance on Crafter, achieving a mean score of 19.4 that substantially improves on the previous high score of 14.5 by DreamerV3 with uniform replay, while also maintaining similar performance on the Deepmind Control Suite. Code for Curious Replay is available at https://github.com/AutonomousAgentsLab/curiousreplay