The Value-Improvement Path: Towards Better Representations for Reinforcement Learning
This work addresses the problem of improving representation learning for reinforcement learning agents, particularly in complex environments like Atari games, by introducing a novel perspective on value prediction as a holistic sequence.
The paper tackles the challenge of representation learning in value-based reinforcement learning by proposing to approximate the entire sequence of value functions (value-improvement path) holistically, rather than focusing on the current policy's value function. This approach, tested on Atari 2600 games, resulted in the augmented agent achieving approximately double the mean and median performance compared to the baseline.
In value-based reinforcement learning (RL), unlike in supervised learning, the agent faces not a single, stationary, approximation problem, but a sequence of value prediction problems. Each time the policy improves, the nature of the problem changes, shifting both the distribution of states and their values. In this paper we take a novel perspective, arguing that the value prediction problems faced by an RL agent should not be addressed in isolation, but rather as a single, holistic, prediction problem. An RL algorithm generates a sequence of policies that, at least approximately, improve towards the optimal policy. We explicitly characterize the associated sequence of value functions and call it the value-improvement path. Our main idea is to approximate the value-improvement path holistically, rather than to solely track the value function of the current policy. Specifically, we discuss the impact that this holistic view of RL has on representation learning. We demonstrate that a representation that spans the past value-improvement path will also provide an accurate value approximation for future policy improvements. We use this insight to better understand existing approaches to auxiliary tasks and to propose new ones. To test our hypothesis empirically, we augmented a standard deep RL agent with an auxiliary task of learning the value-improvement path. In a study of Atari 2600 games, the augmented agent achieved approximately double the mean and median performance of the baseline agent.