Adversarial Intrinsic Motivation for Reinforcement Learning
This addresses the challenge of efficient exploration in goal-conditioned RL for robotics, though it is incremental as it builds on existing techniques like HER.
The paper tackled the problem of goal-conditioned reinforcement learning by proposing a method to estimate the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution, using it as a supplemental reward function to direct exploration. The result showed that combining this approach with Hindsight Experience Replay significantly accelerated learning on simulated robotics tasks compared to other exploration methods.
Learning with an objective to minimize the mismatch with a reference distribution has been shown to be useful for generative modeling and imitation learning. In this paper, we investigate whether one such objective, the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution, can be utilized effectively for reinforcement learning (RL) tasks. Specifically, this paper focuses on goal-conditioned reinforcement learning where the idealized (unachievable) target distribution has full measure at the goal. This paper introduces a quasimetric specific to Markov Decision Processes (MDPs) and uses this quasimetric to estimate the above Wasserstein-1 distance. It further shows that the policy that minimizes this Wasserstein-1 distance is the policy that reaches the goal in as few steps as possible. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function. Our experiments show that this reward function changes smoothly with respect to transitions in the MDP and directs the agent's exploration to find the goal efficiently. Additionally, we combine AIM with Hindsight Experience Replay (HER) and show that the resulting algorithm accelerates learning significantly on several simulated robotics tasks when compared to other rewards that encourage exploration or accelerate learning.