Grounded Vision-Language Interpreter for Integrated Task and Motion Planning
It addresses the problem of enabling verifiable and interpretable autonomous robot behaviors for real-world deployment, particularly in manipulation tasks like cooking, but is incremental as it builds on existing vision-language interpreters and planning methods.
The paper tackles the lack of safety and interpretability in vision-language model-based robot planners by proposing ViLaIn-TAMP, a hybrid framework that integrates a vision-language interpreter with a task and motion planning system and a corrective module, achieving an 18% improvement in mean success rate over a baseline and a 32% boost with the corrective module.
While recent advances in vision-language models have accelerated the development of language-guided robot planners, their black-box nature often lacks safety guarantees and interpretability crucial for real-world deployment. Conversely, classical symbolic planners offer rigorous safety verification but require significant expert knowledge for setup. To bridge the current gap, this paper proposes ViLaIn-TAMP, a hybrid planning framework for enabling verifiable, interpretable, and autonomous robot behaviors. ViLaIn-TAMP comprises three main components: (1) a Vision-Language Interpreter (ViLaIn) adapted from previous work that converts multimodal inputs into structured problem specifications, (2) a modular Task and Motion Planning (TAMP) system that grounds these specifications in actionable trajectory sequences through symbolic and geometric constraint reasoning, and (3) a corrective planning (CP) module which receives concrete feedback on failed solution attempts and feed them with constraints back to ViLaIn to refine the specification. We design challenging manipulation tasks in a cooking domain and evaluate our framework. Experimental results demonstrate that ViLaIn-TAMP outperforms a VLM-as-a-planner baseline by 18% in mean success rate, and that adding the CP module boosts mean success rate by 32%.