ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activities
This work provides a new benchmark for embodied AI agents to reason about human activities, but it is incremental as it builds on existing datasets and methods.
The authors introduced ViLPAct, a benchmark for multimodal human activity planning, addressing the challenge of compositional generalization in forecasting future actions from video and text. They created a dataset of 2.9k videos with crowdsourced intents and tested four baselines, finding key difficulties in generalization and multimodal integration.
We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from \charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods. According to our extensive experiments, the key challenges are compositional generalization and effective use of information from both modalities.