Regularized Behavior Value Estimation
This addresses a key challenge in offline RL for real-world applications where data is limited, though it appears incremental as it builds on existing methods with specific improvements.
The paper tackles the problem of overestimation errors in offline reinforcement learning by introducing Regularized Behavior Value Estimation (R-BVE), which estimates behavior policy values during training and uses ranking regularization, achieving state-of-the-art performance on the RL Unplugged ATARI dataset and outperforming other methods on new datasets.
Offline reinforcement learning restricts the learning process to rely only on logged-data without access to an environment. While this enables real-world applications, it also poses unique challenges. One important challenge is dealing with errors caused by the overestimation of values for state-action pairs not well-covered by the training data. Due to bootstrapping, these errors get amplified during training and can lead to divergence, thereby crippling learning. To overcome this challenge, we introduce Regularized Behavior Value Estimation (R-BVE). Unlike most approaches, which use policy improvement during training, R-BVE estimates the value of the behavior policy during training and only performs policy improvement at deployment time. Further, R-BVE uses a ranking regularisation term that favours actions in the dataset that lead to successful outcomes. We provide ample empirical evidence of R-BVE's effectiveness, including state-of-the-art performance on the RL Unplugged ATARI dataset. We also test R-BVE on new datasets, from bsuite and a challenging DeepMind Lab task, and show that R-BVE outperforms other state-of-the-art discrete control offline RL methods.