The Value of Reward Lookahead in Reinforcement Learning
This work addresses the problem of leveraging advance reward knowledge in RL for applications like finance and autonomous systems, though it is incremental as it builds on competitive analysis.
The paper quantifies the advantage of having partial future reward information in reinforcement learning by deriving worst-case ratios between standard agents and those with lookahead, relating these ratios to offline RL and reward-free exploration concepts.
In reinforcement learning (RL), agents sequentially interact with changing environments while aiming to maximize the obtained rewards. Usually, rewards are observed only after acting, and so the goal is to maximize the expected cumulative reward. Yet, in many practical settings, reward information is observed in advance -- prices are observed before performing transactions; nearby traffic information is partially known; and goals are oftentimes given to agents prior to the interaction. In this work, we aim to quantifiably analyze the value of such future reward information through the lens of competitive analysis. In particular, we measure the ratio between the value of standard RL agents and that of agents with partial future-reward lookahead. We characterize the worst-case reward distribution and derive exact ratios for the worst-case reward expectations. Surprisingly, the resulting ratios relate to known quantities in offline RL and reward-free exploration. We further provide tight bounds for the ratio given the worst-case dynamics. Our results cover the full spectrum between observing the immediate rewards before acting to observing all the rewards before the interaction starts.