LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agents
This work addresses the challenge of creating interpretable and robust embodied agents for vision-and-language interaction tasks, though it is incremental as it builds on existing benchmarks and methods.
The paper tackles the problem of building an embodied agent that can perform household tasks by understanding natural language instructions, using the ALFRED benchmark, and proposes the LEBP framework with a two-stream process to generate expectations and bind them to actions, achieving comparable performance to SOTA methods with reduced decay in unseen scenarios.
People always desire an embodied agent that can perform a task by understanding language instruction. Moreover, they also want to monitor and expect agents to understand commands the way they expected. But, how to build such an embodied agent is still unclear. Recently, people can explore this problem with the Vision-and-Language Interaction benchmark ALFRED, which requires an agent to perform complicated daily household tasks following natural language instructions in unseen scenes. In this paper, we propose LEBP -- Language Expectation and Binding Policy Module to tackle the ALFRED. The LEBP contains a two-stream process: 1) It first conducts a language expectation module to generate an expectation describing how to perform tasks by understanding the language instruction. The expectation consists of a sequence of sub-steps for the task (e.g., Pick an apple). The expectation allows people to access and check the understanding results of instructions before the agent takes actual actions, in case the task might go wrong. 2) Then, it uses the binding policy module to bind sub-steps in expectation to actual actions to specific scenarios. Actual actions include navigation and object manipulation. Experimental results suggest our approach achieves comparable performance to currently published SOTA methods and can avoid large decay from seen scenarios to unseen scenarios.