Learning Joint Representations of Videos and Sentences with Web Image Search
This addresses video understanding and retrieval for applications like search engines or multimedia analysis, but it is incremental as it builds on existing embedding methods with web image search for disambiguation.
The paper tackles video retrieval from natural language queries and sentence retrieval/description generation from videos by embedding visual and textual inputs into a common space, showing clear improvement in retrieval tasks and comparable performance in description generation.
Our objective is video retrieval based on natural language queries. In addition, we consider the analogous problem of retrieving sentences or generating descriptions given an input video. Recent work has addressed the problem by embedding visual and textual inputs into a common space where semantic similarities correlate to distances. We also adopt the embedding approach, and make the following contributions: First, we utilize web image search in sentence embedding process to disambiguate fine-grained visual concepts. Second, we propose embedding models for sentence, image, and video inputs whose parameters are learned simultaneously. Finally, we show how the proposed model can be applied to description generation. Overall, we observe a clear improvement over the state-of-the-art methods in the video and sentence retrieval tasks. In description generation, the performance level is comparable to the current state-of-the-art, although our embeddings were trained for the retrieval tasks.