Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment
This note clarifies a previously overlooked requirement for practitioners deploying MORL agents with non-linear utility functions.
The paper identifies that multi-objective reinforcement learning with augmented states requires the reward signal to be available after deployment, even if no further learning occurs, which has practical repercussions for real-world applications.
This research note identifies a previously overlooked distinction between multi-objective reinforcement learning (MORL), and more conventional single-objective reinforcement learning (RL). It has previously been noted that the optimal policy for an MORL agent with a non-linear utility function is required to be conditioned on both the current environmental state and on some measure of the previously accrued reward. This is generally implemented by concatenating the observed state of the environment with the discounted sum of previous rewards to create an augmented state. While augmented states have been widely-used in the MORL literature, one implication of their use has not previously been reported -- namely that they require the agent to have continued access to the reward signal (or a proxy thereof) after deployment, even if no further learning is required. This note explains why this is the case, and considers the practical repercussions of this requirement.