Model-based Offline Reinforcement Learning with Local Misspecification
This work addresses offline RL for safe policy improvement in scenarios with imperfect dynamics models, though it is incremental as it builds on existing lower bound methods.
The paper tackles the problem of offline reinforcement learning with model misspecification by proposing a model-based policy performance lower bound and an algorithm for optimal policy selection, achieving competitive performance on D4RL tasks.
We present a model-based offline reinforcement learning policy performance lower bound that explicitly captures dynamics model misspecification and distribution mismatch and we propose an empirical algorithm for optimal offline policy selection. Theoretically, we prove a novel safe policy improvement theorem by establishing pessimism approximations to the value function. Our key insight is to jointly consider selecting over dynamics models and policies: as long as a dynamics model can accurately represent the dynamics of the state-action pairs visited by a given policy, it is possible to approximate the value of that particular policy. We analyze our lower bound in the LQR setting and also show competitive performance to previous lower bounds on policy selection across a set of D4RL tasks.