Experience is the Best Teacher: Grounding VLMs for Robotics through Self-Generated Memory
This addresses the problem of adapting VLMs for robotics in real-world settings, offering a novel approach with significant performance gains, though it is incremental in building on existing VLM and retrieval-augmented generation methods.
The paper tackles the challenge of grounding vision-language models (VLMs) to diverse real-world robots by introducing ExpTeach, a framework that uses self-generated memory of experiences to improve planning and adaptation. It shows that reflection increases success rates from 36% to 84% on four tasks, and long-term memory boosts rates from 22% to 80% across 12 scenarios, including unseen ones.
Vision-language models (VLMs) have been widely adopted in robotics to enable autonomous planning. However, grounding VLMs, originally trained on internet data, to diverse real-world robots remains a challenge. This paper presents ExpTeach, a framework that grounds VLMs to physical robots by building a self-generated memory of real-world experiences. In ExpTeach, the VLM autonomously plans actions, verifies outcomes, reflects on failures, and adapts robot behaviors in a closed loop. The self-generated experiences during this process are then summarized into a long-term memory, enabling retrieval of learned knowledge to guide future tasks via retrieval-augmented generation (RAG). Additionally, ExpTeach enhances the spatial understanding of VLMs with an on-demand image annotation module. In experiments, we show that reflection improves success rates from 36% to 84% on four challenging robotic tasks and observe the emergence of intelligent object interactions, including creative tool use. Across extensive tests on 12 real-world scenarios (including eight unseen ones), we find that grounding with long-term memory boosts single-trial success rates from 22% to 80%, demonstrating the effectiveness and generalizability of ExpTeach.