Finite-Time Error Analysis of Online Model-Based Q-Learning with a Relaxed Sampling Model
This work addresses the problem of sample efficiency in reinforcement learning for researchers, but it is incremental as it extends existing Q-learning methods to a model-based context.
The paper tackles the sample complexity of Q-learning in a model-based framework, finding conditions where it improves sample efficiency over model-free Q-learning through theoretical and empirical analysis.
Reinforcement learning has witnessed significant advancements, particularly with the emergence of model-based approaches. Among these, $Q$-learning has proven to be a powerful algorithm in model-free settings. However, the extension of $Q$-learning to a model-based framework remains relatively unexplored. In this paper, we delve into the sample complexity of $Q$-learning when integrated with a model-based approach. Through theoretical analyses and empirical evaluations, we seek to elucidate the conditions under which model-based $Q$-learning excels in terms of sample efficiency compared to its model-free counterpart.