On The Effect of Auxiliary Tasks on Representation Dynamics
This work addresses a fundamental gap in reinforcement learning theory for researchers and practitioners, but it appears incremental as it builds on existing understanding of auxiliary tasks.
The paper tackled the problem of understanding how auxiliary tasks affect representation learning in reinforcement learning by analyzing temporal difference algorithms, establishing a connection between transition operator spectral decomposition and representations, and applied these insights to improve auxiliary task selection in sparse-reward environments, resulting in unspecified performance gains.
While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments.