LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning
This addresses the problem of improving procedural planning in embodied AI for researchers and developers, though it appears incremental by building on existing vision-language models.
The paper tackles multimodal procedural planning for embodied AI systems by introducing LLaPa, a vision-language model framework that integrates textual task descriptions and visual environmental images to generate executable action sequences, with results showing it outperforms advanced models on benchmarks like ActPlan-1K and ALFRED with superior LCS and correctness scores.
While large language models (LLMs) have advanced procedural planning for embodied AI systems through strong reasoning abilities, the integration of multimodal inputs and counterfactual reasoning remains underexplored. To tackle these challenges, we introduce LLaPa, a vision-language model framework designed for multimodal procedural planning. LLaPa generates executable action sequences from textual task descriptions and visual environmental images using vision-language models (VLMs). Furthermore, we enhance LLaPa with two auxiliary modules to improve procedural planning. The first module, the Task-Environment Reranker (TER), leverages task-oriented segmentation to create a task-sensitive feature space, aligning textual descriptions with visual environments and emphasizing critical regions for procedural execution. The second module, the Counterfactual Activities Retriever (CAR), identifies and emphasizes potential counterfactual conditions, enhancing the model's reasoning capability in counterfactual scenarios. Extensive experiments on ActPlan-1K and ALFRED benchmarks demonstrate that LLaPa generates higher-quality plans with superior LCS and correctness, outperforming advanced models. The code and models are available https://github.com/sunshibo1234/LLaPa.