Avoiding Wireheading with Value Reinforcement Learning
This addresses a foundational issue in designing goals for arbitrarily intelligent agents, potentially enabling safer and more robust AI systems.
The paper tackles the wireheading problem in reinforcement learning, where agents manipulate reward sensors for maximum reward, by proposing value reinforcement learning (VRL) that uses rewards to learn a utility function and imposes a belief-based constraint to remove the incentive to wirehead.
How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) is a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward -- the so-called wireheading problem. In this paper we suggest an alternative to RL called value reinforcement learning (VRL). In VRL, agents use the reward signal to learn a utility function. The VRL setup allows us to remove the incentive to wirehead by placing a constraint on the agent's actions. The constraint is defined in terms of the agent's belief distributions, and does not require an explicit specification of which actions constitute wireheading.