Diverse and Coherent Paragraph Generation from Images
This work improves paragraph generation for applications like video summarization and accessibility, though it is incremental as it builds on existing techniques.
The paper tackles the problem of generating diverse and coherent paragraphs from images, addressing limitations of traditional image captioning methods. It proposes a novel approach using coherence vectors, global topic vectors, and a variational auto-encoder, achieving state-of-the-art results on two datasets.
Paragraph generation from images, which has gained popularity recently, is an important task for video summarization, editing, and support of the disabled. Traditional image captioning methods fall short on this front, since they aren't designed to generate long informative descriptions. Moreover, the vanilla approach of simply concatenating multiple short sentences, possibly synthesized from a classical image captioning system, doesn't embrace the intricacies of paragraphs: coherent sentences, globally consistent structure, and diversity. To address those challenges, we propose to augment paragraph generation techniques with 'coherence vectors', 'global topic vectors', and modeling of the inherent ambiguity of associating paragraphs with images, via a variational auto-encoder formulation. We demonstrate the effectiveness of the developed approach on two datasets, outperforming existing state-of-the-art techniques on both.