Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes
This work addresses robustness in offline RL for high-dimensional environments, providing a sample-efficient solution with concrete improvements over existing methods.
The paper tackles the sample complexity of offline distributionally robust linear Markov Decision Processes to address robustness in high-dimensional spaces, achieving a sample complexity bound that outperforms prior methods by at least O~(d) where d is the feature dimension.
In offline reinforcement learning (RL), the absence of active exploration calls for attention on the model robustness to tackle the sim-to-real gap, where the discrepancy between the simulated and deployed environments can significantly undermine the performance of the learned policy. To endow the learned policy with robustness in a sample-efficient manner in the presence of high-dimensional state-action space, this paper considers the sample complexity of distributionally robust linear Markov decision processes (MDPs) with an uncertainty set characterized by the total variation distance using offline data. We develop a pessimistic model-based algorithm and establish its sample complexity bound under minimal data coverage assumptions, which outperforms prior art by at least $\widetilde{O}(d)$, where $d$ is the feature dimension. We further improve the performance guarantee of the proposed algorithm by incorporating a carefully-designed variance estimator.