Rethinking the Form of Latent States in Image Captioning
This work addresses a specific bottleneck in image captioning models, offering incremental improvements in performance and interpretability.
The paper tackled the problem of latent state representation in image captioning by proposing two-dimensional maps instead of vectors, achieving higher performance on MSCOCO and Flickr30k datasets with comparable parameter sizes.
RNNs and their variants have been widely adopted for image captioning. In RNNs, the production of a caption is driven by a sequence of latent states. Existing captioning models usually represent latent states as vectors, taking this practice for granted. We rethink this choice and study an alternative formulation, namely using two-dimensional maps to encode latent states. This is motivated by the curiosity about a question: how the spatial structures in the latent states affect the resultant captions? Our study on MSCOCO and Flickr30k leads to two significant observations. First, the formulation with 2D states is generally more effective in captioning, consistently achieving higher performance with comparable parameter sizes. Second, 2D states preserve spatial locality. Taking advantage of this, we visually reveal the internal dynamics in the process of caption generation, as well as the connections between input visual domain and output linguistic domain.