TEACh: Task-driven Embodied Agents that Chat
This addresses the need for embodied agents to communicate effectively in human environments, though it is incremental as it focuses on dataset creation and initial benchmarks.
The authors tackled the problem of enabling robots to interact via natural language in household tasks by introducing TEACh, a dataset of over 3,000 human-human dialogues in simulation, and proposed benchmarks to evaluate models on dialogue understanding, language grounding, and task execution.
Robots operating in human spaces must be able to engage in natural language interaction with people, both understanding and executing instructions, and using conversation to resolve ambiguity and recover from mistakes. To study this, we introduce TEACh, a dataset of over 3,000 human--human, interactive dialogues to complete household tasks in simulation. A Commander with access to oracle information about a task communicates in natural language with a Follower. The Follower navigates through and interacts with the environment to complete tasks varying in complexity from "Make Coffee" to "Prepare Breakfast", asking questions and getting additional information from the Commander. We propose three benchmarks using TEACh to study embodied intelligence challenges, and we evaluate initial models' abilities in dialogue understanding, language grounding, and task execution.