Understanding Guided Image Captioning Performance across Domains
This work is significant for users who need image captioning models to provide more focused and user-interest-aware descriptions, rather than generic ones. It is an incremental step towards more controllable image captioning.
This paper addresses the limitation of image captioning models in incorporating user interest by introducing a method that uses guiding text to control the focus of image captions. They found that models trained on Conceptual Captions generalize better to out-of-domain images and guiding texts compared to those trained on Visual Genome, highlighting the importance of large, unrestricted-domain datasets and style diversity for in-the-wild guided image captioning.
Image captioning models generally lack the capability to take into account user interest, and usually default to global descriptions that try to balance readability, informativeness, and information overload. On the other hand, VQA models generally lack the ability to provide long descriptive answers, while expecting the textual question to be quite precise. We present a method to control the concepts that an image caption should focus on, using an additional input called the guiding text that refers to either groundable or ungroundable concepts in the image. Our model consists of a Transformer-based multimodal encoder that uses the guiding text together with global and object-level image features to derive early-fusion representations used to generate the guided caption. While models trained on Visual Genome data have an in-domain advantage of fitting well when guided with automatic object labels, we find that guided captioning models trained on Conceptual Captions generalize better on out-of-domain images and guiding texts. Our human-evaluation results indicate that attempting in-the-wild guided image captioning requires access to large, unrestricted-domain training datasets, and that increased style diversity (even without increasing the number of unique tokens) is a key factor for improved performance.