What You Say Is What You Show: Visual Narration Detection in Instructional Videos
This addresses data quality issues for researchers and practitioners using instructional videos for learning tasks, but it is incremental as it builds on existing multi-modal and weakly supervised techniques.
The paper tackles the problem of noisy narrations in instructional videos by introducing visual narration detection, a task to determine if narrations match the actions shown, and proposes the WYS^2 method that outperforms baselines and improves video summarization and alignment.
Narrated ''how-to'' videos have emerged as a promising data source for a wide range of learning problems, from learning visual representations to training robot policies. However, this data is extremely noisy, as the narrations do not always describe the actions demonstrated in the video. To address this problem we introduce the novel task of visual narration detection, which entails determining whether a narration is visually depicted by the actions in the video. We propose What You Say is What You Show (WYS^2), a method that leverages multi-modal cues and pseudo-labeling to learn to detect visual narrations with only weakly labeled data. Our model successfully detects visual narrations in in-the-wild videos, outperforming strong baselines, and we demonstrate its impact for state-of-the-art summarization and temporal alignment of instructional videos.