Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
This addresses the challenge of improving offline RL for domains like robotics and autonomous driving by leveraging readily available video data, though it appears incremental as it builds on existing model-based approaches.
The paper tackles the problem of offline reinforcement learning struggling with suboptimal behaviors and inaccurate value estimation by introducing VeoRL, a model-based method that uses unlabeled video data to build an interactive world model, achieving performance gains of over 100% in some cases across visual control tasks.
Offline reinforcement learning (RL) enables policy optimization using static datasets, avoiding the risks and costs of extensive real-world exploration. However, it struggles with suboptimal offline behaviors and inaccurate value estimation due to the lack of environmental interaction. We present Video-Enhanced Offline RL (VeoRL), a model-based method that constructs an interactive world model from diverse, unlabeled video data readily available online. Leveraging model-based behavior guidance, our approach transfers commonsense knowledge of control policy and physical dynamics from natural videos to the RL agent within the target domain. VeoRL achieves substantial performance gains (over 100% in some cases) across visual control tasks in robotic manipulation, autonomous driving, and open-world video games.