Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
This addresses the challenge of offline RL for scenarios where agents must learn from static datasets without active interaction, though it appears incremental as it builds on existing methods by focusing on states rather than actions.
The paper tackles the problem of distributional shift in offline reinforcement learning by proposing a Pessimistic Offline Reinforcement Learning (PessORL) algorithm that penalizes high values at out-of-distribution states to ensure the learned value function lower bounds the true value, resulting in better performance on benchmark tasks compared to methods only handling out-of-distribution actions.
Reinforcement Learning (RL) has been shown effective in domains where the agent can learn policies by actively interacting with its operating environment. However, if we change the RL scheme to offline setting where the agent can only update its policy via static datasets, one of the major issues in offline reinforcement learning emerges, i.e. distributional shift. We propose a Pessimistic Offline Reinforcement Learning (PessORL) algorithm to actively lead the agent back to the area where it is familiar by manipulating the value function. We focus on problems caused by out-of-distribution (OOD) states, and deliberately penalize high values at states that are absent in the training dataset, so that the learned pessimistic value function lower bounds the true value anywhere within the state space. We evaluate the PessORL algorithm on various benchmark tasks, where we show that our method gains better performance by explicitly handling OOD states, when compared to those methods merely considering OOD actions.