Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspective
This work provides a theoretical foundation for a key technique in multi-goal RL, which is incremental but clarifies connections to imitation learning.
The paper tackles the problem of understanding hindsight goal relabeling in multi-goal reinforcement learning by recasting it as a divergence minimization problem within an imitation learning framework, leading to insights such as Q-learning outperforming behavioral cloning and selective application of BC loss improving performance.
Hindsight goal relabeling has become a foundational technique in multi-goal reinforcement learning (RL). The essential idea is that any trajectory can be seen as a sub-optimal demonstration for reaching its final state. Intuitively, learning from those arbitrary demonstrations can be seen as a form of imitation learning (IL). However, the connection between hindsight goal relabeling and imitation learning is not well understood. In this paper, we propose a novel framework to understand hindsight goal relabeling from a divergence minimization perspective. Recasting the goal reaching problem in the IL framework not only allows us to derive several existing methods from first principles, but also provides us with the tools from IL to improve goal reaching algorithms. Experimentally, we find that under hindsight relabeling, Q-learning outperforms behavioral cloning (BC). Yet, a vanilla combination of both hurts performance. Concretely, we see that the BC loss only helps when selectively applied to actions that get the agent closer to the goal according to the Q-function. Our framework also explains the puzzling phenomenon wherein a reward of (-1, 0) results in significantly better performance than a (0, 1) reward for goal reaching.