A Geometric Perspective on Self-Supervised Policy Adaptation
This work addresses the challenge of adapting RL agents to unpredictable distractions without rewards, which is crucial for real-world applications, though it builds incrementally on prior self-supervised methods.
The paper tackles the problem of long-term self-supervised policy adaptation in reinforcement learning under real-world distractions, revealing undesirable effects in embedding spaces and proposing geometric manipulations to improve generalization, with empirical results showing performance gains.
One of the most challenging aspects of real-world reinforcement learning (RL) is the multitude of unpredictable and ever-changing distractions that could divert an agent from what was tasked to do in its training environment. While an agent could learn from reward signals to ignore them, the complexity of the real-world can make rewards hard to acquire, or, at best, extremely sparse. A recent class of self-supervised methods have shown promise that reward-free adaptation under challenging distractions is possible. However, previous work focused on a short one-episode adaptation setting. In this paper, we consider a long-term adaptation setup that is more akin to the specifics of the real-world and propose a geometric perspective on self-supervised adaptation. We empirically describe the processes that take place in the embedding space during this adaptation process, reveal some of its undesirable effects on performance and show how they can be eliminated. Moreover, we theoretically study how actor-based and actor-free agents can further generalise to the target environment by manipulating the geometry of the manifolds described by the actor and critic functions.