Multimodal Contextualized Plan Prediction for Embodied Task Completion
This work addresses the challenge of making task planning more transferable to physical robots in embodied AI, but it is incremental as it builds on existing datasets and methods.
The paper tackled the problem of predicting high-level plans for embodied task completion from natural language, using multimodal context to improve plan quality, and found that plan prediction and execution modules are interdependent, with oracle plans revealing room for improvement in models.
Task planning is an important component of traditional robotics systems enabling robots to compose fine grained skills to perform more complex tasks. Recent work building systems for translating natural language to executable actions for task completion in simulated embodied agents is focused on directly predicting low level action sequences that would be expected to be directly executable by a physical robot. In this work, we instead focus on predicting a higher level plan representation for one such embodied task completion dataset - TEACh, under the assumption that techniques for high-level plan prediction from natural language are expected to be more transferable to physical robot systems. We demonstrate that better plans can be predicted using multimodal context, and that plan prediction and plan execution modules are likely dependent on each other and hence it may not be ideal to fully decouple them. Further, we benchmark execution of oracle plans to quantify the scope for improvement in plan prediction models.