Controllable Image Captioning via Prompting
This addresses the problem of generating stylized captions for users in image captioning, though it is incremental as it builds on existing pre-trained models with prompt-based fine-tuning.
The paper tackles the lack of controllability in image captioning by introducing a unified model that generates captions in diverse styles (e.g., rough, detailed, factual, emotional) using prompt learning, achieving outstanding performance on benchmarks like COCO Karpathy split and TextCaps.
Despite the remarkable progress of image captioning, existing captioners typically lack the controllable capability to generate desired image captions, e.g., describing the image in a rough or detailed manner, in a factual or emotional view, etc. In this paper, we show that a unified model is qualified to perform well in diverse domains and freely switch among multiple styles. Such a controllable capability is achieved by embedding the prompt learning into the image captioning framework. To be specific, we design a set of prompts to fine-tune the pre-trained image captioner. These prompts allow the model to absorb stylized data from different domains for joint training, without performance degradation in each domain. Furthermore, we optimize the prompts with learnable vectors in the continuous word embedding space, avoiding the heuristic prompt engineering and meanwhile exhibiting superior performance. In the inference stage, our model is able to generate desired stylized captions by choosing the corresponding prompts. Extensive experiments verify the controllable capability of the proposed method. Notably, we achieve outstanding performance on two diverse image captioning benchmarks including COCO Karpathy split and TextCaps using a unified model.