Towards A Unified Agent with Foundation Models
This work proposes a unified framework for RL agents using foundation models, potentially benefiting robotics and AI by simplifying algorithm design, though it appears incremental in integrating existing models.
The paper tackles the problem of embedding language and vision-language models into reinforcement learning agents to address fundamental RL challenges like exploration and data reuse, demonstrating substantial performance improvements in a robotic manipulation environment.
Language Models and Vision Language Models have recently demonstrated unprecedented capabilities in terms of understanding human intentions, reasoning, scene understanding, and planning-like behaviour, in text form, among many others. In this work, we investigate how to embed and leverage such abilities in Reinforcement Learning (RL) agents. We design a framework that uses language as the core reasoning tool, exploring how this enables an agent to tackle a series of fundamental RL challenges, such as efficient exploration, reusing experience data, scheduling skills, and learning from observations, which traditionally require separate, vertically designed algorithms. We test our method on a sparse-reward simulated robotic manipulation environment, where a robot needs to stack a set of objects. We demonstrate substantial performance improvements over baselines in exploration efficiency and ability to reuse data from offline datasets, and illustrate how to reuse learned skills to solve novel tasks or imitate videos of human experts.