Corruption-Robust Offline Reinforcement Learning
This addresses the robustness of offline RL for applications where data integrity is critical, such as safety-sensitive domains, and highlights it as a strictly harder problem than online RL in this context.
The paper tackles the problem of adversarial corruption in offline reinforcement learning, where an adversary can modify a fraction of the dataset, and shows that a worst-case optimality gap of Ω(dε) is unavoidable in linear MDPs, while proposing robust algorithms that achieve near-matching performances with and without full data coverage.
We study the adversarial robustness in offline reinforcement learning. Given a batch dataset consisting of tuples $(s, a, r, s')$, an adversary is allowed to arbitrarily modify $ε$ fraction of the tuples. From the corrupted dataset the learner aims to robustly identify a near-optimal policy. We first show that a worst-case $Ω(dε)$ optimality gap is unavoidable in linear MDP of dimension $d$, even if the adversary only corrupts the reward element in a tuple. This contrasts with dimension-free results in robust supervised learning and best-known lower-bound in the online RL setting with corruption. Next, we propose robust variants of the Least-Square Value Iteration (LSVI) algorithm utilizing robust supervised learning oracles, which achieve near-matching performances in cases both with and without full data coverage. The algorithm requires the knowledge of $ε$ to design the pessimism bonus in the no-coverage case. Surprisingly, in this case, the knowledge of $ε$ is necessary, as we show that being adaptive to unknown $ε$ is impossible.This again contrasts with recent results on corruption-robust online RL and implies that robust offline RL is a strictly harder problem.