3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model
This work addresses the problem of generating stylized captions for images, which is incremental as it builds on existing captioning methods by incorporating multi-modality features.
The paper tackles stylish image captioning by proposing the 3M model, which uses multi-modality features and a Multi-UPDOWN architecture to generate human-like captions, achieving competitive performance on datasets like PERSONALITY-CAPTIONS and FlickrStyle10K as measured by metrics such as BLEU and CIDEr.
In this paper, we build a multi-style generative model for stylish image captioning which uses multi-modality image features, ResNeXt features and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decode them to captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITY-CAPTIONS dataset and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc. A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions.