Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
This work addresses the lack of theoretical results for offline RL with neural networks, offering foundational insights for researchers in reinforcement learning and machine learning theory.
The paper tackles the problem of offline reinforcement learning with deep ReLU networks by establishing a sample complexity bound of n = ̃O(H^{4 + 4d/α} κ_μ^{1 + d/α} ε^{-2 - 2d/α}), which holds under novel conditions like Besov dynamic closure and correlated structure, providing the first theoretical characterization beyond linear regimes.
Offline reinforcement learning (RL) leverages previously collected data for policy optimization without any further active exploration. Despite the recent interest in this problem, its theoretical results in neural network function approximation settings remain elusive. In this paper, we study the statistical theory of offline RL with deep ReLU network function approximation. In particular, we establish the sample complexity of $n = \tilde{\mathcal{O}}( H^{4 + 4 \frac{d}α} κ_μ^{1 + \frac{d}α} ε^{-2 - 2\frac{d}α} )$ for offline RL with deep ReLU networks, where $κ_μ$ is a measure of distributional shift, {$H = (1-γ)^{-1}$ is the effective horizon length}, $d$ is the dimension of the state-action space, $α$ is a (possibly fractional) smoothness parameter of the underlying Markov decision process (MDP), and $ε$ is a user-specified error. Notably, our sample complexity holds under two novel considerations: the Besov dynamic closure and the correlated structure. While the Besov dynamic closure subsumes the dynamic conditions for offline RL in the prior works, the correlated structure renders the prior works of offline RL with general/neural network function approximation improper or inefficient {in long (effective) horizon problems}. To the best of our knowledge, this is the first theoretical characterization of the sample complexity of offline RL with deep neural network function approximation under the general Besov regularity condition that goes beyond {the linearity regime} in the traditional Reproducing Hilbert kernel spaces and Neural Tangent Kernels.