Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning
For researchers in meta-reinforcement learning, this work provides a novel solution to distribution shifts and sparse-reward challenges, enabling more robust generalization to unseen environments.
The paper addresses context and policy distribution shifts in offline meta-reinforcement learning, particularly under sparse rewards, by proposing a framework that combines information-theoretic task representation learning with a Transformer-based world model. The method achieves superior stability and generalization, outperforming state-of-the-art approaches in out-of-distribution and sparse-reward settings.
Offline meta-reinforcement learning leverages static datasets to enable agents to generalize to unseen environments by combining offline efficiency with meta-learning adaptability, yet it faces key challenges from context and policy distribution shifts. These issues hinder agents from adapting to online environments, and are further exacerbated under sparse-reward settings. As a result, agents often become trapped in an inherent pattern dilemma, failing to achieve robust generalization. In this work, we propose a novel framework that integrates information-theoretic task representation learning with a Transformer-based stochastic world model. Our approach extracts task-defining latent variables that are invariant to behavior policy, thereby effectively mitigating the context distribution shift. To further handle policy shift and model exploitation, we apply a conservative value penalty to imagination-based rollouts, preventing the policy from exploiting model inaccuracies while maintaining robust adaptation. Extensive evaluations demonstrate that our method outperforms state-of-the-art approaches, with superior stability and generalization under out-of-distribution and sparse-reward settings.