Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
This addresses the challenge of domain adaptation for reinforcement learning agents in potentially expensive target environments, though it appears incremental as it builds on existing unsupervised skill discovery methods.
The paper tackles the problem of transferring skills learned in a source environment to a target environment with different dynamics in unsupervised reinforcement learning, proposing a method that uses dynamics-aware rewards to achieve smooth deployment in the target.
Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuming, limiting the rollout in some potentially expensive target environments. The intuitive approach of training in another interaction-rich environment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce a KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments to demonstrate that our method can effectively learn skills that can be smoothly deployed in target.