On Value Functions and the Agent-Environment Boundary
This work addresses a foundational theoretical issue in reinforcement learning that affects the consistency of optimality analyses across different problem formulations.
The paper tackles the problem of non-unique definitions of value functions in reinforcement learning due to arbitrary agent-environment boundaries, which complicates theoretical optimality guarantees. It proposes a boundary-invariant analysis for Fitted Q-Iteration, ensuring assumptions and guarantees are independent of boundary choices, and extends the discussion to related issues like state resetting and imitation learning.
When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary, causing problems in theoretical analyses that provide optimality guarantees. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting and Monte-Carlo Tree Search, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data.