Conceptual Reinforcement Learning for Language-Conditioned Tasks
This work addresses the problem of poor generalization in language-conditioned reinforcement learning for AI agents, offering an incremental improvement over existing methods.
The paper tackles the challenge of transferring and adapting reinforcement learning policies to unseen environments by proposing a conceptual reinforcement learning (CRL) framework that learns concept-like joint representations from language-conditioned tasks, resulting in up to 70% improvement in training efficiency and up to 30% improvement in generalization ability.
Despite the broad application of deep reinforcement learning (RL), transferring and adapting the policy to unseen but similar environments is still a significant challenge. Recently, the language-conditioned policy is proposed to facilitate policy transfer through learning the joint representation of observation and text that catches the compact and invariant information across environments. Existing studies of language-conditioned RL methods often learn the joint representation as a simple latent layer for the given instances (episode-specific observation and text), which inevitably includes noisy or irrelevant information and cause spurious correlations that are dependent on instances, thus hurting generalization performance and training efficiency. To address this issue, we propose a conceptual reinforcement learning (CRL) framework to learn the concept-like joint representation for language-conditioned policy. The key insight is that concepts are compact and invariant representations in human cognition through extracting similarities from numerous instances in real-world. In CRL, we propose a multi-level attention encoder and two mutual information constraints for learning compact and invariant concepts. Verified in two challenging environments, RTFM and Messenger, CRL significantly improves the training efficiency (up to 70%) and generalization ability (up to 30%) to the new environment dynamics.