Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
This work addresses the challenge of sparse rewards in goal-conditioned RL, which is crucial for applications like robotics, but appears incremental as it builds on existing hindsight techniques.
The authors tackled the problem of goal-conditioned reinforcement learning with sparse rewards by proposing a graphical model framework with an EM algorithm, which significantly outperformed model-free baselines on various benchmarks.
We propose a graphical model framework for goal-conditioned RL, with an EM algorithm that operates on the lower bound of the RL objective. The E-step provides a natural interpretation of how 'learning in hindsight' techniques, such as HER, to handle extremely sparse goal-conditioned rewards. The M-step reduces policy optimization to supervised learning updates, which greatly stabilizes end-to-end training on high-dimensional inputs such as images. We show that the combined algorithm, hEM significantly outperforms model-free baselines on a wide range of goal-conditioned benchmarks with sparse rewards.