Empirical Analysis of Image Caption Generation using Deep Learning
This is an incremental study for researchers in computer vision and NLP, focusing on empirical comparisons of existing methods.
The paper tackled image caption generation by experimenting with various CNN encoders and attention-based LSTM decoders, analyzing performance using metrics like BLEU and CIDEr, and exploring explainability with visual attention maps.
Automated image captioning is one of the applications of Deep Learning which involves fusion of work done in computer vision and natural language processing, and it is typically performed using Encoder-Decoder architectures. In this project, we have implemented and experimented with various flavors of multi-modal image captioning networks where ResNet101, DenseNet121 and VGG19 based CNN Encoders and Attention based LSTM Decoders were explored. We have studied the effect of beam size and the use of pretrained word embeddings and compared them to baseline CNN encoder and RNN decoder architecture. The goal is to analyze the performance of each approach using various evaluation metrics including BLEU, CIDEr, ROUGE and METEOR. We have also explored model explainability using Visual Attention Maps (VAM) to highlight parts of the images which has maximum contribution for predicting each word of the generated caption.