Explore and Control with Adversarial Surprise
This addresses the problem of extracting meaningful behaviors without reward engineering for researchers and practitioners in unsupervised RL, representing a novel method rather than an incremental improvement.
The paper tackles the challenge of unsupervised reinforcement learning in high-dimensional, stochastic environments by proposing an adversarial method where two policies compete over observation entropy, driving exploration and control. The result shows that this approach maximizes state coverage in block MDPs, outperforms state-of-the-art methods in exploration and zero-shot transfer, and leads to the emergence of complex skills.
Unsupervised reinforcement learning (RL) studies how to leverage environment statistics to learn useful behaviors without the cost of reward engineering. However, a central challenge in unsupervised RL is to extract behaviors that meaningfully affect the world and cover the range of possible outcomes, without getting distracted by inherently unpredictable, uncontrollable, and stochastic elements in the environment. To this end, we propose an unsupervised RL method designed for high-dimensional, stochastic environments based on an adversarial game between two policies (which we call Explore and Control) controlling a single body and competing over the amount of observation entropy the agent experiences. The Explore agent seeks out states that maximally surprise the Control agent, which in turn aims to minimize surprise, and thereby manipulate the environment to return to familiar and predictable states. The competition between these two policies drives them to seek out increasingly surprising parts of the environment while learning to gain mastery over them. We show formally that the resulting algorithm maximizes coverage of the underlying state in block MDPs with stochastic observations, providing theoretical backing to our hypothesis that this procedure avoids uncontrollable and stochastic distractions. Our experiments further demonstrate that Adversarial Surprise leads to the emergence of complex and meaningful skills, and outperforms state-of-the-art unsupervised reinforcement learning methods in terms of both exploration and zero-shot transfer to downstream tasks.