Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text
This work addresses video description generation for AI applications, but it is incremental as it builds on existing LSTM methods with added linguistic features.
The paper tackled the problem of generating natural language descriptions for videos by integrating linguistic knowledge from large text corpora into an LSTM-based architecture, resulting in significant improvements in grammaticality and modest gains in descriptive quality on YouTube and movie datasets.
This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant improvements in grammaticality while modestly improving descriptive quality.