iParaphrasing: Extracting Visually Grounded Paraphrases via an Image
This addresses the need for diverse phrasal expressions in multimodal tasks like visual question answering and image captioning, but it is an incremental step as it builds on existing paraphrase and multimodal research.
The paper tackles the problem of extracting visually grounded paraphrases (VGPs) from images, proposing the iParaphrasing task and reporting initial results using various methods, including a novel neural network with image attention.
A paraphrase is a restatement of the meaning of a text in other words. Paraphrases have been studied to enhance the performance of many natural language processing tasks. In this paper, we propose a novel task iParaphrasing to extract visually grounded paraphrases (VGPs), which are different phrasal expressions describing the same visual concept in an image. These extracted VGPs have the potential to improve language and image multimodal tasks such as visual question answering and image captioning. How to model the similarity between VGPs is the key of iParaphrasing. We apply various existing methods as well as propose a novel neural network-based method with image attention, and report the results of the first attempt toward iParaphrasing.