Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
This work addresses robustness in offline RL for real-world applications where data corruption is common, representing an incremental improvement over existing methods.
The paper tackles the problem of offline reinforcement learning (RL) performance degradation due to noisy or maliciously corrupted datasets, finding that implicit Q-learning (IQL) shows resilience but suffers from heavy-tail Q-function targets under dynamics corruption, and proposes Robust IQL (RIQL) which demonstrates highly robust performance in diverse corruption scenarios.
Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios.