Memory-Attended Recurrent Network for Video Captioning
This work addresses the problem of generating more comprehensive video captions for applications like accessibility and content indexing, though it appears incremental as it builds on existing encoder-decoder frameworks with a memory enhancement.
The paper tackles the limitation of typical encoder-decoder video captioning models in capturing multiple visual contexts for words across training videos by proposing a Memory-Attended Recurrent Network (MARN), which achieves higher captioning quality and outperforms state-of-the-art methods on two real-world datasets.
Typical techniques for video captioning follow the encoder-decoder framework, which can only focus on one source video being processed. A potential disadvantage of such design is that it cannot capture the multiple visual context information of a word appearing in more than one relevant videos in training data. To tackle this limitation, we propose the Memory-Attended Recurrent Network (MARN) for video captioning, in which a memory structure is designed to explore the full-spectrum correspondence between a word and its various similar visual contexts across videos in training data. Thus, our model is able to achieve a more comprehensive understanding for each word and yield higher captioning quality. Furthermore, the built memory structure enables our method to model the compatibility between adjacent words explicitly instead of asking the model to learn implicitly, as most existing models do. Extensive validation on two real-word datasets demonstrates that our MARN consistently outperforms state-of-the-art methods.