Residuals-based Offline Reinforcement Learning
This work addresses offline RL challenges for high-stakes applications, but it appears incremental as it builds on existing methods with a novel twist.
The paper tackles the problem of distribution shift and restrictive data coverage assumptions in offline reinforcement learning by proposing a residuals-based framework that incorporates estimation error into policy optimization, demonstrating effectiveness in a stochastic CartPole environment.
Offline reinforcement learning (RL) has received increasing attention for learning policies from previously collected data without interaction with the real environment, which is particularly important in high-stakes applications. While a growing body of work has developed offline RL algorithms, these methods often rely on restrictive assumptions about data coverage and suffer from distribution shift. In this paper, we propose a residuals-based offline RL framework for general state and action spaces. Specifically, we define a residuals-based Bellman optimality operator that explicitly incorporates estimation error in learning transition dynamics into policy optimization by leveraging empirical residuals. We show that this Bellman operator is a contraction mapping and identify conditions under which its fixed point is asymptotically optimal and possesses finite-sample guarantees. We further develop a residuals-based offline deep Q-learning (DQN) algorithm. Using a stochastic CartPole environment, we demonstrate the effectiveness of our residuals-based offline DQN algorithm.