Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning
This addresses the challenge of exploration in reinforcement learning for researchers and practitioners, offering a scalable, plug-and-play solution that is incremental over existing world model frameworks.
The paper tackles efficient exploration in sparse-reward reinforcement learning by introducing Optimistic World Models (OWMs), which incorporate optimism into model learning via a gradient-based loss, resulting in significant improvements in sample efficiency and cumulative return compared to baseline methods.
Efficient exploration remains a central challenge in reinforcement learning (RL), particularly in sparse-reward environments. We introduce Optimistic World Models (OWMs), a principled and scalable framework for optimistic exploration that brings classical reward-biased maximum likelihood estimation (RBMLE) from adaptive control into deep RL. In contrast to upper confidence bound (UCB)-style exploration methods, OWMs incorporate optimism directly into model learning by augmentation with an optimistic dynamics loss that biases imagined transitions toward higher-reward outcomes. This fully gradient-based loss requires neither uncertainty estimates nor constrained optimization. Our approach is plug-and-play with existing world model frameworks, preserving scalability while requiring only minimal modifications to standard training procedures. We instantiate OWMs within two state-of-the-art world model architectures, leading to Optimistic DreamerV3 and Optimistic STORM, which demonstrate significant improvements in sample efficiency and cumulative return compared to their baseline counterparts.