Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models
This work addresses the problem of improving robotic manipulation for AI and robotics researchers by providing an automated feedback method, though it is incremental as it builds on preference-based policy learning and multimodal LLMs.
The paper tackled the challenge of aligning natural language instructions from LLMs with robotic operations by using a multimodal LLM, CriticGPT, to provide preference feedback from image inputs for robot manipulation tasks, resulting in effective generalization to new tasks and surpassing state-of-the-art pre-trained representation models in guiding policy learning on Meta-World tasks.
Recently, there has been considerable attention towards leveraging large language models (LLMs) to enhance decision-making processes. However, aligning the natural language text instructions generated by LLMs with the vectorized operations required for execution presents a significant challenge, often necessitating task-specific details. To circumvent the need for such task-specific granularity, inspired by preference-based policy learning approaches, we investigate the utilization of multimodal LLMs to provide automated preference feedback solely from image inputs to guide decision-making. In this study, we train a multimodal LLM, termed CriticGPT, capable of understanding trajectory videos in robot manipulation tasks, serving as a critic to offer analysis and preference feedback. Subsequently, we validate the effectiveness of preference labels generated by CriticGPT from a reward modeling perspective. Experimental evaluation of the algorithm's preference accuracy demonstrates its effective generalization ability to new tasks. Furthermore, performance on Meta-World tasks reveals that CriticGPT's reward model efficiently guides policy learning, surpassing rewards based on state-of-the-art pre-trained representation models.