Off-Policy Reinforcement Learning with High Dimensional Reward
This work provides a foundational advancement for researchers in reinforcement learning, enabling handling of complex reward structures, though it is incremental in extending distributional RL theory.
The paper tackles the problem of reinforcement learning with high-dimensional rewards by establishing robust theoretical foundations for distributional RL, proving the contraction property of the Bellman operator in infinite-dimensional Banach spaces and showing effective approximation in lower-dimensional Euclidean spaces, leading to a novel algorithm that addresses previously intractable problems.
Conventional off-policy reinforcement learning (RL) focuses on maximizing the expected return of scalar rewards. Distributional RL (DRL), in contrast, studies the distribution of returns with the distributional Bellman operator in a Euclidean space, leading to highly flexible choices for utility. This paper establishes robust theoretical foundations for DRL. We prove the contraction property of the Bellman operator even when the reward space is an infinite-dimensional separable Banach space. Furthermore, we demonstrate that the behavior of high- or infinite-dimensional returns can be effectively approximated using a lower-dimensional Euclidean space. Leveraging these theoretical insights, we propose a novel DRL algorithm that tackles problems which have been previously intractable using conventional reinforcement learning approaches.