Meta-Learning to Explore via Memory Density Feedback
This addresses exploration challenges in reinforcement learning for agents operating in complex environments, though it appears incremental as it builds on existing intrinsic reward methods with a meta-learning twist.
The paper tackles the problem of exploration in reinforcement learning by introducing a meta-learning approach where the agent learns to minimize the probability density of new observations relative to its memories, enabling it to maximize exploration progress in novel states within a single episode.
Exploration algorithms for reinforcement learning typically replace or augment the reward function with an additional ``intrinsic'' reward that trains the agent to seek previously unseen states of the environment. Here, we consider an exploration algorithm that exploits meta-learning, or learning to learn, such that the agent learns to maximize its exploration progress within a single episode, even between epochs of training. The agent learns a policy that aims to minimize the probability density of new observations with respect to all of its memories. In addition, it receives as feedback evaluations of the current observation density and retains that feedback in a recurrent network. By remembering trajectories of density, the agent learns to navigate a complex and growing landscape of familiarity in real-time, allowing it to maximize its exploration progress even in completely novel states of the environment for which its policy has not been trained.