Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Function
This addresses a specific technical issue in meta-gradient RL that can cause catastrophic failure, making it an incremental improvement for researchers in reinforcement learning.
The paper identifies a bias in meta-gradient reinforcement learning that arises from using the meta-learned discount factor for advantage estimation in the outer objective, which can lead to myopic policies, and proposes a solution by adding a second critic head trained with the outer loss discount factor, demonstrating improved performance in experiments.
Meta-gradient Reinforcement Learning (RL) allows agents to self-tune their hyper-parameters in an online fashion during training. In this paper, we identify a bias in the meta-gradient of current meta-gradient RL approaches. This bias comes from using the critic that is trained using the meta-learned discount factor for the advantage estimation in the outer objective which requires a different discount factor. Because the meta-learned discount factor is typically lower than the one used in the outer objective, the resulting bias can cause the meta-gradient to favor myopic policies. We propose a simple solution to this issue: we eliminate this bias by using an alternative, \emph{outer} value function in the estimation of the outer loss. To obtain this outer value function we add a second head to the critic network and train it alongside the classic critic, using the outer loss discount factor. On an illustrative toy problem, we show that the bias can cause catastrophic failure of current meta-gradient RL approaches, and show that our proposed solution fixes it. We then apply our method to a more complex environment and demonstrate that fixing the meta-gradient bias can significantly improve performance.