Contrastive Unsupervised Learning of World Model with Invariant Causal Features
This work addresses the challenge of robust and generalizable world models in reinforcement learning, particularly for out-of-distribution and sim-to-real scenarios, representing a strong specific gain rather than a foundational advancement.
The paper tackles the problem of learning world models with invariant causal features for reinforcement learning, using contrastive unsupervised learning and an intervention invariant auxiliary task. The result is a method that significantly outperforms state-of-the-art approaches on out-of-distribution navigation tasks in iGibson, excels in sim-to-real transfer, and performs competitively on the DeepMind control suite.
In this paper we present a world model, which learns causal features using the invariance principle. In particular, we use contrastive unsupervised learning to learn the invariant causal features, which enforces invariance across augmentations of irrelevant parts or styles of the observation. The world-model-based reinforcement learning methods independently optimize representation learning and the policy. Thus naive contrastive loss implementation collapses due to a lack of supervisory signals to the representation learning module. We propose an intervention invariant auxiliary task to mitigate this issue. Specifically, we utilize depth prediction to explicitly enforce the invariance and use data augmentation as style intervention on the RGB observation space. Our design leverages unsupervised representation learning to learn the world model with invariant causal features. Our proposed method significantly outperforms current state-of-the-art model-based and model-free reinforcement learning methods on out-of-distribution point navigation tasks on the iGibson dataset. Moreover, our proposed model excels at the sim-to-real transfer of our perception learning module. Finally, we evaluate our approach on the DeepMind control suite and enforce invariance only implicitly since depth is not available. Nevertheless, our proposed model performs on par with the state-of-the-art counterpart.