Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
This work addresses the challenge of developing AI agents that can adapt to diverse, unseen tasks using offline data, providing insights for researchers in offline RL and planning, but it is incremental as it systematically evaluates existing paradigms.
The paper tackled the problem of comparing reinforcement learning and control-based methods for solving tasks from reward-free offline data, finding that model-free RL performs best with large, high-quality datasets while model-based planning generalizes better to unseen environments and is more data-efficient.
A long-standing goal in AI is to develop agents capable of solving diverse tasks across a range of environments, including those never seen during training. Two dominant paradigms address this challenge: (i) reinforcement learning (RL), which learns policies via trial and error, and (ii) optimal control, which plans actions using a known or learned dynamics model. However, their comparative strengths in the offline setting - where agents must learn from reward-free trajectories - remain underexplored. In this work, we systematically evaluate RL and control-based methods on a suite of navigation tasks, using offline datasets of varying quality. On the RL side, we consider goal-conditioned and zero-shot methods. On the control side, we train a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning. We investigate how factors such as data diversity, trajectory quality, and environment variability influence the performance of these approaches. Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts and is more data-efficient, while achieving trajectory stitching performance comparable to leading model-free methods. Notably, planning with a latent dynamics model proves to be a strong approach for handling suboptimal offline data and adapting to diverse environments.