Near-Optimal Sample Complexity in Reward-Free Kernel-Based Reinforcement Learning
This work addresses the statistical efficiency problem for researchers and practitioners in reinforcement learning, particularly in nonlinear function approximation, and is incremental by building on prior work with less restrictive assumptions.
The paper tackles the problem of determining the sample complexity for designing near-optimal policies in reward-free reinforcement learning with kernel-based models, achieving near-optimal sample complexity under a generative model and extending it to more general settings with an additional factor of episode length.
Reinforcement Learning (RL) problems are being considered under increasingly more complex structures. While tabular and linear models have been thoroughly explored, the analytical study of RL under nonlinear function approximation, especially kernel-based models, has recently gained traction for their strong representational capacity and theoretical tractability. In this context, we examine the question of statistical efficiency in kernel-based RL within the reward-free RL framework, specifically asking: how many samples are required to design a near-optimal policy? Existing work addresses this question under restrictive assumptions about the class of kernel functions. We first explore this question by assuming a generative model, then relax this assumption at the cost of increasing the sample complexity by a factor of H, the length of the episode. We tackle this fundamental problem using a broad class of kernels and a simpler algorithm compared to prior work. Our approach derives new confidence intervals for kernel ridge regression, specific to our RL setting, which may be of broader applicability. We further validate our theoretical findings through simulations.