VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning
This work addresses the challenge of improving model accuracy in offline RL for applications where online interactions are risky or costly, representing an incremental advancement by integrating value inconsistency feedback into existing methods.
The paper tackles the problem of unreliable conservatism in model-based offline reinforcement learning due to model errors by introducing VIPO, which uses self-supervised feedback from value estimation to enhance model training, achieving state-of-the-art performance on nearly all tasks in D4RL and NeoRL benchmarks.
Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the one estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. In particular, it achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. Overall, VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy.