Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
This work improves robustness in offline MBRL for data-driven control applications, though it is incremental as it builds on existing two-stage methods.
The paper tackles the problem of offline model-based reinforcement learning (MBRL) by addressing the objective mismatch between world model learning and policy optimization, which leads to non-robust policies vulnerable to adversarial noise. It proposes a framework that dynamically adapts the world model with the policy under a unified objective, achieving state-of-the-art performance on twelve D4RL MuJoCo and three Tokamak Control tasks.
Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential generalization beyond the dataset support. However, most existing offline MBRL methods follow a two-stage training procedure: first learning a world model by maximizing the likelihood of the observed transitions, then optimizing a policy to maximize its expected return under the learned model. This objective mismatch results in a world model that is not necessarily optimized for effective policy learning. Moreover, we observe that policies learned via offline MBRL often lack robustness during deployment, and small adversarial noise in the environment can lead to significant performance degradation. To address these, we propose a framework that dynamically adapts the world model alongside the policy under a unified learning objective aimed at improving robustness. At the core of our method is a maximin optimization problem, which we solve by innovatively utilizing Stackelberg learning dynamics. We provide theoretical analysis to support our design and introduce computationally efficient implementations. We benchmark our algorithm on twelve noisy D4RL MuJoCo tasks and three stochastic Tokamak Control tasks, demonstrating its state-of-the-art performance.