What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?
This addresses a design choice in neural image captioning systems, offering an incremental improvement for researchers and practitioners in computer vision and natural language processing.
The paper compared two architectures for using recurrent neural networks (RNNs) in image captioning: one where RNNs generate captions by injecting image features, and another where RNNs encode linguistic features with late merging of image features. It found that late merging generally outperforms injection, indicating RNNs are better as encoders than generators.
In neural image captioning systems, a recurrent neural network (RNN) is typically viewed as the primary `generation' component. This view suggests that the image features should be `injected' into the RNN. This is in fact the dominant view in the literature. Alternatively, the RNN can instead be viewed as only encoding the previously generated words. This view suggests that the RNN should only be used to encode linguistic features and that only the final representation should be `merged' with the image features at a later stage. This paper compares these two architectures. We find that, in general, late merging outperforms injection, suggesting that RNNs are better viewed as encoders, rather than generators.