What Can Learned Intrinsic Rewards Capture?
This work addresses the challenge of designing reward functions in RL, offering a method to learn them adaptively, though it is incremental as it builds on existing meta-learning and intrinsic reward concepts.
The paper tackles the problem of learning intrinsic reward functions in reinforcement learning, proposing a meta-gradient framework that captures knowledge about long-term exploration and exploitation, and shows that these learned rewards can generalize to different agents and environmental dynamics.
The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and immutable. In this paper, we instead consider the proposition that the reward function itself can be a good locus of learned knowledge. To investigate this, we propose a scalable meta-gradient framework for learning useful intrinsic reward functions across multiple lifetimes of experience. Through several proof-of-concept experiments, we show that it is feasible to learn and capture knowledge about long-term exploration and exploitation into a reward function. Furthermore, we show that unlike policy transfer methods that capture "how" the agent should behave, the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment by capturing "what" the agent should strive to do.