Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models
This work addresses the challenge of language-assisted teleoperation for human-robot cooperation, though it appears incremental as it builds on existing foundation models for a specific application.
The paper tackles the problem of enabling robots to interpret verbal instructions for placing support contacts to prevent falls during manipulation, and it demonstrates the method's effectiveness in real-world experiments with a humanoid robot, showing that users can quickly learn to achieve accurate contact placements.
This paper presents Words2Contact, a language-guided multi-contact placement pipeline leveraging large language models and vision language models. Our method is a key component for language-assisted teleoperation and human-robot cooperation, where human operators can instruct the robots where to place their support contacts before whole-body reaching or manipulation using natural language. Words2Contact transforms the verbal instructions of a human operator into contact placement predictions; it also deals with iterative corrections, until the human is satisfied with the contact location identified in the robot's field of view. We benchmark state-of-the-art LLMs and VLMs for size and performance in contact prediction. We demonstrate the effectiveness of the iterative correction process, showing that users, even naive, quickly learn how to instruct the system to obtain accurate locations. Finally, we validate Words2Contact in real-world experiments with the Talos humanoid robot, instructed by human operators to place support contacts on different locations and surfaces to avoid falling when reaching for distant objects.