Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding
This addresses the problem of limited multilingual resources for vision-language story understanding, providing a dataset to advance research in this domain, though it is incremental as it builds on existing datasets like SyMoN.
The authors tackled the scarcity of manually annotated video-text correspondence and English bias in story video-text alignment by constructing M-SYMON, a large-scale multilingual dataset with 13,166 movie summary videos in 7 languages and 101.5 hours of annotated video, which improved SOTA methods by 15.7 and 16.2 percentage points on Clip Accuracy and Sentence IoU scores.
Story video-text alignment, a core task in computational story understanding, aims to align video clips with corresponding sentences in their descriptions. However, progress on the task has been held back by the scarcity of manually annotated video-text correspondence and the heavy concentration on English narrations of Hollywood movies. To address these issues, in this paper, we construct a large-scale multilingual video story dataset named Multilingual Synopses of Movie Narratives (M-SYMON), containing 13,166 movie summary videos from 7 languages, as well as manual annotation of fine-grained video-text correspondences for 101.5 hours of video. Training on the human annotated data from SyMoN outperforms the SOTA methods by 15.7 and 16.2 percentage points on Clip Accuracy and Sentence IoU scores, respectively, demonstrating the effectiveness of the annotations. As benchmarks for future research, we create 6 baseline approaches with different multilingual training strategies, compare their performance in both intra-lingual and cross-lingual setups, exemplifying the challenges of multilingual video-text alignment. The dataset is released at: https://github.com/insundaycathy/M-SyMoN