NewsStories: Illustrating articles with visual summaries
This addresses the challenge of generating visual summaries for news articles, which is an incremental improvement over existing methods by handling loose illustrative correspondence and varying text lengths.
The paper tackled the problem of learning self-supervised visual-language representations for tasks requiring multiple images and long text narratives, such as illustrating news articles, and introduced a large-scale dataset with over 31M articles and 22M images, showing that a baseline method outperforms state-of-the-art by 10% on zero-shot image-set retrieval.
Recent self-supervised approaches have used large-scale image-text datasets to learn powerful representations that transfer to many tasks without finetuning. These methods often assume that there is one-to-one correspondence between its images and their (short) captions. However, many tasks require reasoning about multiple images and long text narratives, such as describing news articles with visual summaries. Thus, we explore a novel setting where the goal is to learn a self-supervised visual-language representation that is robust to varying text length and the number of images. In addition, unlike prior work which assumed captions have a literal relation to the image, we assume images only contain loose illustrative correspondence with the text. To explore this problem, we introduce a large-scale multimodal dataset containing over 31M articles, 22M images and 1M videos. We show that state-of-the-art image-text alignment methods are not robust to longer narratives with multiple images. Finally, we introduce an intuitive baseline that outperforms these methods on zero-shot image-set retrieval by 10% on the GoodNews dataset.