OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
This work addresses the problem of enabling mobile manipulators to generalize to open-ended instructions and environments, which is incremental as it builds on existing navigation, manipulation, and vision models.
The paper tackles the challenge of open-world mobile manipulation by proposing a multi-modal agent architecture with an agentic data synthesis pipeline, achieving state-of-the-art performance compared to other foundation models like GPT-4o and demonstrating strong zero-shot generalization in real-world experiments.
The rapid progress of navigation, manipulation, and vision models has made mobile manipulators capable in many specialized tasks. However, the open-world mobile manipulation (OWMM) task remains a challenge due to the need for generalization to open-ended instructions and environments, as well as the systematic complexity to integrate high-level decision making with low-level robot control based on both global scene understanding and current agent state. To address this complexity, we propose a novel multi-modal agent architecture that maintains multi-view scene frames and agent states for decision-making and controls the robot by function calling. A second challenge is the hallucination from domain shift. To enhance the agent performance, we further introduce an agentic data synthesis pipeline for the OWMM task to adapt the VLM model to our task domain with instruction fine-tuning. We highlight our fine-tuned OWMM-VLM as the first dedicated foundation model for mobile manipulators with global scene understanding, robot state tracking, and multi-modal action generation in a unified model. Through experiments, we demonstrate that our model achieves SOTA performance compared to other foundation models including GPT-4o and strong zero-shot generalization in real world. The project page is at https://github.com/HHYHRHY/OWMM-Agent