Information-Theoretic Considerations in Batch Reinforcement Learning
This work addresses foundational theoretical gaps in batch RL, which is incremental as it revisits existing assumptions rather than introducing new methods.
The paper tackles the problem of understanding the necessity and naturalness of assumptions in batch reinforcement learning for value-function approximation, providing theoretical results to deepen the understanding of these methods.
Value-function approximation methods that operate in batch mode have foundational importance to reinforcement learning (RL). Finite sample guarantees for these methods often crucially rely on two types of assumptions: (1) mild distribution shift, and (2) representation conditions that are stronger than realizability. However, the necessity ("why do we need them?") and the naturalness ("when do they hold?") of such assumptions have largely eluded the literature. In this paper, we revisit these assumptions and provide theoretical results towards answering the above questions, and make steps towards a deeper understanding of value-function approximation.