Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference
This provides a resource for researchers in multimodal AI to test inference systems on semantically rich human actions, though it is incremental as it builds on existing dataset efforts.
The authors tackled the lack of datasets for multimodal logical inference by creating a new video-and-language dataset with human actions, resulting in 200 videos, 5,554 action labels, and 1,942 action triplets for evaluating systems handling complex sentences like negation and quantification.
This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form <subject, predicate, object> that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.