Egocentric Vision Language Planning
This addresses the challenge of building more general embodied agents for household tasks, though it appears incremental by integrating existing techniques like diffusion models and LMMs.
The paper tackles the problem of grounding large multi-modal models (LMMs) in the physical world for long-horizon tasks from an egocentric perspective in household scenarios, proposing EgoPlan, which improves task success rates compared to baselines.
We explore leveraging large multi-modal models (LMMs) and text2image models to build a more general embodied agent. LMMs excel in planning long-horizon tasks over symbolic abstractions but struggle with grounding in the physical world, often failing to accurately identify object positions in images. A bridge is needed to connect LMMs to the physical world. The paper proposes a novel approach, egocentric vision language planning (EgoPlan), to handle long-horizon tasks from an egocentric perspective in varying household scenarios. This model leverages a diffusion model to simulate the fundamental dynamics between states and actions, integrating techniques like style transfer and optical flow to enhance generalization across different environmental dynamics. The LMM serves as a planner, breaking down instructions into sub-goals and selecting actions based on their alignment with these sub-goals, thus enabling more generalized and effective decision-making. Experiments show that EgoPlan improves long-horizon task success rates from the egocentric view compared to baselines across household scenarios.