Foundation Models for Semantic Novelty in Reinforcement Learning
This addresses the problem of effective exploration in RL for agents in procedurally-generated environments, representing an incremental improvement by applying existing foundation models to a known bottleneck.
The paper tackles the challenge of exploration in reinforcement learning by introducing an intrinsic reward based on pre-trained CLIP embeddings, which drives exploration towards semantically meaningful states and outperforms state-of-the-art methods in sparse-reward procedurally-generated environments.
Effectively exploring the environment is a key challenge in reinforcement learning (RL). We address this challenge by defining a novel intrinsic reward based on a foundation model, such as contrastive language image pretraining (CLIP), which can encode a wealth of domain-independent semantic visual-language knowledge about the world. Specifically, our intrinsic reward is defined based on pre-trained CLIP embeddings without any fine-tuning or learning on the target RL task. We demonstrate that CLIP-based intrinsic rewards can drive exploration towards semantically meaningful states and outperform state-of-the-art methods in challenging sparse-reward procedurally-generated environments.