MAGNet: Multi-Region Attention-Assisted Grounding of Natural Language Queries at Phrase Level
This work addresses the challenge of accurately localizing objects in images based on textual queries, which is important for applications like visual search and human-computer interaction, though it is incremental as it builds on existing spatial attention networks.
The paper tackled the problem of grounding natural language queries at the phrase level in images, achieving over 12% improvement over state-of-the-art on the ReferIt dataset and competitive results on Flickr30k Entities without using additional context or attributes.
Grounding free-form textual queries necessitates an understanding of these textual phrases and its relation to the visual cues to reliably reason about the described locations. Spatial attention networks are known to learn this relationship and focus its gaze on salient objects in the image. Thus, we propose to utilize spatial attention networks for image-level visual-textual fusion preserving local (word) and global (phrase) information to refine region proposals with an in-network Region Proposal Network (RPN) and detect single or multiple regions for a phrase query. We focus only on the phrase query - ground truth pair (referring expression) for a model independent of the constraints of the datasets i.e. additional attributes, context etc. For such referring expression dataset ReferIt game, our Multi-region Attention-assisted Grounding network (MAGNet) achieves over 12\% improvement over the state-of-the-art. Without the context from image captions and attribute information in Flickr30k Entities, we still achieve competitive results compared to the state-of-the-art.