How2: A Large-scale Dataset for Multimodal Language Understanding
This dataset addresses the need for multimodal resources in language processing research, though it is incremental as it builds on existing data collection efforts.
The authors introduced How2, a large-scale multimodal dataset of instructional videos with English subtitles and Portuguese translations, and provided baseline models for tasks like machine translation and multimodal summarization to facilitate research in multimodal language understanding.
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.