"Let's not Quote out of Context": Unified Vision-Language Pretraining for Context Assisted Image Captioning
This addresses the need for automated, context-sensitive captioning in marketing and enterprise settings, though it is incremental as it builds on existing OFA models.
The paper tackles the problem of generating context-aware image captions for enterprise content by proposing a unified vision-language model based on OFA, which incorporates contextual information during pretraining. It achieves state-of-the-art results with an improvement of up to 8.34 CIDEr score on benchmark datasets.
Well-formed context aware image captions and tags in enterprise content such as marketing material are critical to ensure their brand presence and content recall. Manual creation and updates to ensure the same is non trivial given the scale and the tedium towards this task. We propose a new unified Vision-Language (VL) model based on the One For All (OFA) model, with a focus on context-assisted image captioning where the caption is generated based on both the image and its context. Our approach aims to overcome the context-independent (image and text are treated independently) nature of the existing approaches. We exploit context by pretraining our model with datasets of three tasks: news image captioning where the news article is the context, contextual visual entailment, and keyword extraction from the context. The second pretraining task is a new VL task, and we construct and release two datasets for the task with 1.1M and 2.2K data instances. Our system achieves state-of-the-art results with an improvement of up to 8.34 CIDEr score on the benchmark news image captioning datasets. To the best of our knowledge, ours is the first effort at incorporating contextual information in pretraining the models for the VL tasks.