Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning
This work addresses the challenge of improving image captioning accuracy for applications like assistive technology and content indexing, though it appears incremental as it builds on existing attention-based encoder-decoder frameworks.
The paper tackled the problem of generating fine-grained image captions by proposing a stacked cross-modal feature consolidation (SCFC) attention network that integrates high-level semantic concepts and visual information, achieving state-of-the-art performance on MSCOCO and Flickr30K datasets.
Recently, the attention-enriched encoder-decoder framework has aroused great interest in image captioning due to its overwhelming progress. Many visual attention models directly leverage meaningful regions to generate image descriptions. However, seeking a direct transition from visual space to text is not enough to generate fine-grained captions. This paper exploits a feature-compounding approach to bring together high-level semantic concepts and visual information regarding the contextual environment fully end-to-end. Thus, we propose a stacked cross-modal feature consolidation (SCFC) attention network for image captioning in which we simultaneously consolidate cross-modal features through a novel compounding function in a multi-step reasoning fashion. Besides, we jointly employ spatial information and context-aware attributes (CAA) as the principal components in our proposed compounding function, where our CAA provides a concise context-sensitive semantic representation. To make better use of consolidated features potential, we further propose an SCFC-LSTM as the caption generator, which can leverage discriminative semantic information through the caption generation process. The experimental results indicate that our proposed SCFC can outperform various state-of-the-art image captioning benchmarks in terms of popular metrics on the MSCOCO and Flickr30K datasets.