Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
This work addresses the challenge of scaling reinforcement learning to large state spaces without online interactions, which is crucial for applications like robotics and healthcare, but it is primarily theoretical and incremental in nature.
The paper tackles the problem of offline reinforcement learning in large state spaces by learning policies from historical data without online interactions, introducing key concepts like expressivity assumptions and data coverage to provide a theoretical framework with sample and computational complexity guarantees.
This article introduces the theory of offline reinforcement learning in large state spaces, where good policies are learned from historical data without online interactions with the environment. Key concepts introduced include expressivity assumptions on function approximation (e.g., Bellman completeness vs. realizability) and data coverage (e.g., all-policy vs. single-policy coverage). A rich landscape of algorithms and results is described, depending on the assumptions one is willing to make and the sample and computational complexity guarantees one wishes to achieve. We also discuss open questions and connections to adjacent areas.