Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning
This work addresses visual storytelling for applications like automated content creation, though it appears incremental as it builds on existing foundation models with novel training components.
The paper tackles the problem of generating coherent multi-sentence stories from image sequences by proposing a framework that leverages pretrained foundation models with a lightweight vision-language mapping network and a multimodal contrastive objective. The result is stories that are diverse, coherent, informative, and interesting, as demonstrated through extensive automatic metrics and human evaluations.
Visual storytelling systems generate multi-sentence stories from image sequences. In this task, capturing contextual information and bridging visual variation bring additional challenges. We propose a simple yet effective framework that leverages the generalization capabilities of pretrained foundation models, only training a lightweight vision-language mapping network to connect modalities, while incorporating context to enhance coherence. We introduce a multimodal contrastive objective that also improves visual relevance and story informativeness. Extensive experimental results, across both automatic metrics and human evaluations, demonstrate that the stories generated by our framework are diverse, coherent, informative, and interesting.