Towards a Better Understanding of Representation Dynamics under TD-learning
This work provides incremental insights into representation learning for RL practitioners, focusing on theoretical analysis and empirical validation in tabular and Atari environments.
The paper investigates how end-to-end TD-learning affects state representation dynamics in reinforcement learning, showing that it reduces value approximation error in reversible environments and linking representation dynamics to spectral decomposition, which supports using multiple value functions as an auxiliary task for representation learning.
TD-learning is a foundation reinforcement learning (RL) algorithm for value prediction. Critical to the accuracy of value predictions is the quality of state representations. In this work, we consider the question: how does end-to-end TD-learning impact the representation over time? Complementary to prior work, we provide a set of analysis that sheds further light on the representation dynamics under TD-learning. We first show that when the environments are reversible, end-to-end TD-learning strictly decreases the value approximation error over time. Under further assumptions on the environments, we can connect the representation dynamics with spectral decomposition over the transition matrix. This latter finding establishes fitting multiple value functions from randomly generated rewards as a useful auxiliary task for representation learning, as we empirically validate on both tabular and Atari game suites.