Discourse Parsing in Videos: A Multi-modal Appraoch
This work addresses the challenge of extracting discourse cues from videos for applications like Visual Dialog and Visual Storytelling, but it is incremental as it builds on text-level discourse parsing by extending it to the visual domain.
The paper tackles the problem of understanding discourse relations between scenes in videos by proposing Visual Discourse Parsing, and it presents a dataset of 310 videos with discourse cues to evaluate their approach, which avoids manual scene identification and annotation.
Text-level discourse parsing aims to unmask how two sentences in the text are related to each other. We propose the task of Visual Discourse Parsing, which requires understanding discourse relations among scenes in a video. Here we use the term scene to refer to a subset of video frames that can better summarize the video. In order to collect a dataset for learning discourse cues from videos, one needs to manually identify the scenes from a large pool of video frames and then annotate the discourse relations between them. This is clearly a time consuming, expensive and tedious task. In this work, we propose an approach to identify discourse cues from the videos without the need to explicitly identify and annotate the scenes. We also present a novel dataset containing 310 videos and the corresponding discourse cues to evaluate our approach. We believe that many of the multi-discipline AI problems such as Visual Dialog and Visual Storytelling would greatly benefit from the use of visual discourse cues.