SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasets
This addresses a key bottleneck in offline RL for real-world applications with high-dimensional visual data, offering a novel solution to improve policy learning from low-quality datasets.
The paper tackles the problem of biased model uncertainty estimation in model-based offline reinforcement learning when observations contain complex distractors, proposing SeMOPO to decompose latent states and estimate uncertainty only on endogenous parts, resulting in substantial outperformance over baseline methods on a new low-quality vision dataset.
Model-based offline reinforcement Learning (RL) is a promising approach that leverages existing data effectively in many real-world applications, especially those involving high-dimensional inputs like images and videos. To alleviate the distribution shift issue in offline RL, existing model-based methods heavily rely on the uncertainty of learned dynamics. However, the model uncertainty estimation becomes significantly biased when observations contain complex distractors with non-trivial dynamics. To address this challenge, we propose a new approach - \emph{Separated Model-based Offline Policy Optimization} (SeMOPO) - decomposing latent states into endogenous and exogenous parts via conservative sampling and estimating model uncertainty on the endogenous states only. We provide a theoretical guarantee of model uncertainty and performance bound of SeMOPO. To assess the efficacy, we construct the Low-Quality Vision Deep Data-Driven Datasets for RL (LQV-D4RL), where the data are collected by non-expert policy and the observations include moving distractors. Experimental results show that our method substantially outperforms all baseline methods, and further analytical experiments validate the critical designs in our method. The project website is \href{https://sites.google.com/view/semopo}{https://sites.google.com/view/semopo}.