Reinforcement Learning with Depreciating Assets
This addresses a novel scenario in reinforcement learning for agents dealing with time-sensitive rewards, but it is incremental as it extends existing frameworks with a specific depreciation assumption.
The paper tackles the problem of reinforcement learning where rewards lose value over time after being received, proposing a model of depreciating assets with exponential discounting. It formulates a Bellman equation for optimality and develops a model-free learning approach to find optimal policies.
A basic assumption of traditional reinforcement learning is that the value of a reward does not change once it is received by an agent. The present work forgoes this assumption and considers the situation where the value of a reward decays proportionally to the time elapsed since it was obtained. Emphasizing the inflection point occurring at the time of payment, we use the term asset to refer to a reward that is currently in the possession of an agent. Adopting this language, we initiate the study of depreciating assets within the framework of infinite-horizon quantitative optimization. In particular, we propose a notion of asset depreciation, inspired by classical exponential discounting, where the value of an asset is scaled by a fixed discount factor at each time step after it is obtained by the agent. We formulate a Bellman-style equational characterization of optimality in this context and develop a model-free reinforcement learning approach to obtain optimal policies.