Offline Reinforcement Learning with On-Policy Q-Function Regularization
This addresses the problem of catastrophic extrapolation error in offline RL for researchers and practitioners, representing an incremental improvement over existing regularization approaches.
The paper tackles the challenge of distribution shift in offline reinforcement learning by proposing to regularize towards the Q-function of the behavior policy instead of the policy itself, which is easier to estimate. The resulting algorithms demonstrate strong performance on the D4RL benchmarks.
The core challenge of offline reinforcement learning (RL) is dealing with the (potentially catastrophic) extrapolation error induced by the distribution shift between the history dataset and the desired policy. A large portion of prior work tackles this challenge by implicitly/explicitly regularizing the learning policy towards the behavior policy, which is hard to estimate reliably in practice. In this work, we propose to regularize towards the Q-function of the behavior policy instead of the behavior policy itself, under the premise that the Q-function can be estimated more reliably and easily by a SARSA-style estimate and handles the extrapolation error more straightforwardly. We propose two algorithms taking advantage of the estimated Q-function through regularizations, and demonstrate they exhibit strong performance on the D4RL benchmarks.