Query-guided Regression Network with Context Policy for Phrase Grounding
This work addresses the problem of phrase grounding for computer vision and natural language processing applications, offering a novel method that integrates spatial regression and reinforcement learning to leverage context, representing a strong specific gain rather than a broad paradigm shift.
The paper tackles phrase grounding by localizing objects in images based on textual descriptions, proposing a Query-guided Regression network with Context policy (QRC Net) that jointly learns proposal generation, regression, and context policy networks, resulting in accuracy improvements of 14.25% on Flickr30K Entities and 17.14% on Referit Game over state-of-the-art methods.
Given a textual description of an image, phrase grounding localizes objects in the image referred by query phrases in the description. State-of-the-art methods address the problem by ranking a set of proposals based on the relevance to each query, which are limited by the performance of independent proposal generation systems and ignore useful cues from context in the description. In this paper, we adopt a spatial regression method to break the performance limit, and introduce reinforcement learning techniques to further leverage semantic context information. We propose a novel Query-guided Regression network with Context policy (QRC Net) which jointly learns a Proposal Generation Network (PGN), a Query-guided Regression Network (QRN) and a Context Policy Network (CPN). Experiments show QRC Net provides a significant improvement in accuracy on two popular datasets: Flickr30K Entities and Referit Game, with 14.25% and 17.14% increase over the state-of-the-arts respectively.