Adversarial Semantic Alignment for Improved Image Captions
This work addresses the problem of improving image captioning models for the computer vision community, offering incremental advancements in training methods and evaluation tools.
The paper tackled image captioning by proposing a conditional GAN training approach with a context-aware LSTM captioner and co-attentive discriminator, showing that Self-critical Sequence Training (SCST) outperforms Gumbel Straight-Through in stability and results, and introduced a new semantic score and Out of Context test set for evaluation, with SCST achieving strong performance on benchmarks.
In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions. We empirically focus on the viability of two training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST, even without accessing discriminator gradients directly. We also address the problem of automatic evaluation for captioning models and introduce a new semantic score, and show its correlation to human judgement. As an evaluation paradigm, we argue that an important criterion for a captioner is the ability to generalize to compositions of objects that do not usually co-occur together. To this end, we introduce a small captioned Out of Context (OOC) test set. The OOC set, combined with our semantic score, are the proposed new diagnosis tools for the captioning community. When evaluated on OOC and MS-COCO benchmarks, we show that SCST-based training has a strong performance in both semantic score and human evaluation, promising to be a valuable new approach for efficient discrete GAN training.