R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation
This work addresses the problem of reducing computational waste and improving versatility in MBRL for researchers and practitioners, though it is incremental as it builds on existing frameworks with a novel objective.
The paper tackled the challenge of learning efficient representations in image-based Model-Based Reinforcement Learning by proposing R2-Dreamer, a decoder-free framework that uses a self-supervised redundancy-reduction objective as an internal regularizer, achieving competitive performance with baselines like DreamerV3 while training 1.59x faster and showing substantial gains on tasks with tiny objects.
A central challenge in image-based Model-Based Reinforcement Learning (MBRL) is to learn representations that distill essential information from irrelevant visual details. While promising, reconstruction-based methods often waste capacity on large task-irrelevant regions. Decoder-free methods instead learn robust representations by leveraging Data Augmentation (DA), but reliance on such external regularizers limits versatility. We propose R2-Dreamer, a decoder-free MBRL framework with a self-supervised objective that serves as an internal regularizer, preventing representation collapse without resorting to DA. The core of our method is a redundancy-reduction objective inspired by Barlow Twins, which can be easily integrated into existing frameworks. On DeepMind Control Suite and Meta-World, R2-Dreamer is competitive with strong baselines such as DreamerV3 and TD-MPC2 while training 1.59x faster than DreamerV3, and yields substantial gains on DMC-Subtle with tiny task-relevant objects. These results suggest that an effective internal regularizer can enable versatile, high-performance decoder-free MBRL. Code is available at https://github.com/NM512/r2dreamer.