Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
This work addresses the understudied problem of offline meta-RL, which could enable RL algorithms in real-world applications, though it appears incremental as it builds upon an existing SOTA method.
The paper tackled the challenge of robust task representation learning in offline meta-reinforcement learning by incorporating intra-task attention and inter-task contrastive learning into the FOCAL algorithm, resulting in provably improved performance and robustness across benchmarks.
Meta-learning for offline reinforcement learning (OMRL) is an understudied problem with tremendous potential impact by enabling RL algorithms in many real-world applications. A popular solution to the problem is to infer task identity as augmented state using a context-based encoder, for which efficient learning of robust task representations remains an open challenge. In this work, we provably improve upon one of the SOTA OMRL algorithms, FOCAL, by incorporating intra-task attention mechanism and inter-task contrastive learning objectives, to robustify task representation learning against sparse reward and distribution shift. Theoretical analysis and experiments are presented to demonstrate the superior performance and robustness of our end-to-end and model-free framework compared to prior algorithms across multiple meta-RL benchmarks.