Exploration with Foundation Models: Capabilities, Limitations, and Hybrid Approaches
This work addresses the challenge of exploration in sparse-reward RL for AI researchers, but it is incremental as it focuses on analyzing limitations and a simple hybrid framework.
The paper investigates the use of foundation models for zero-shot exploration in reinforcement learning, finding that while VLMs can infer high-level objectives, they struggle with low-level control, and a hybrid approach improves early-stage sample efficiency in idealized settings.
Exploration in reinforcement learning (RL) remains challenging, particularly in sparse-reward settings. While foundation models possess strong semantic priors, their capabilities as zero-shot exploration agents in classic RL benchmarks are not well understood. We benchmark LLMs and VLMs on multi-armed bandits, Gridworlds, and sparse-reward Atari to test zero-shot exploration. Our investigation reveals a key limitation: while VLMs can infer high-level objectives from visual input, they consistently fail at precise low-level control: the "knowing-doing gap". To analyze a potential bridge for this gap, we investigate a simple on-policy hybrid framework in a controlled, best-case scenario. Our results in this idealized setting show that VLM guidance can significantly improve early-stage sample efficiency, providing a clear analysis of the potential and constraints of using foundation models to guide exploration rather than for end-to-end control.