Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
This work addresses the challenge of textual grounding for applications in human-computer interaction, robotics, and knowledge mining, offering a globally optimal and interpretable solution.
The paper tackled the problem of textual grounding by proposing a unified framework that enables efficient search over all possible bounding boxes, eliminating the need for a first-stage proposal generation. The result was a method that outperformed state-of-the-art approaches by 3.08% on Flickr 30k Entities and 7.77% on ReferItGame datasets.
Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn't rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.