Towards Generating Stylized Image Captions via Adversarial Training
This addresses the issue of limited stylistic diversity in image captioning for applications like creative media, though it is incremental as it builds on existing attention and adversarial methods.
The paper tackled the problem of generating stylized image captions that balance style and content, proposing ATTEND-GAN to improve correlation and variability, resulting in outperforming state-of-the-art models and showing a wider range of stylistic elements in captions.
While most image captioning aims to generate objective descriptions of images, the last few years have seen work on generating visually grounded image captions which have a specific style (e.g., incorporating positive or negative sentiment). However, because the stylistic component is typically the last part of training, current models usually pay more attention to the style at the expense of accurate content description. In addition, there is a lack of variability in terms of the stylistic aspects. To address these issues, we propose an image captioning model called ATTEND-GAN which has two core components: first, an attention-based caption generator to strongly correlate different parts of an image with different parts of a caption; and second, an adversarial training mechanism to assist the caption generator to add diverse stylistic components to the generated captions. Because of these components, ATTEND-GAN can generate correlated captions as well as more human-like variability of stylistic patterns. Our system outperforms the state-of-the-art as well as a collection of our baseline models. A linguistic analysis of the generated captions demonstrates that captions generated using ATTEND-GAN have a wider range of stylistic adjectives and adjective-noun pairs.