Improving One-stage Visual Grounding by Recursive Sub-query Construction
This work improves visual grounding for applications requiring detailed language descriptions, but it is incremental as it builds on existing one-stage methods.
The paper tackles the problem of grounding long and complex queries in one-stage visual grounding by addressing limitations in query modeling, resulting in absolute improvements of 5.0% to 12.8% over state-of-the-art baselines on multiple datasets.
We improve one-stage visual grounding by addressing current limitations on grounding long and complex queries. Existing one-stage methods encode the entire language query as a single sentence embedding vector, e.g., taking the embedding from BERT or the hidden state from LSTM. This single vector representation is prone to overlooking the detailed descriptions in the query. To address this query modeling deficiency, we propose a recursive sub-query construction framework, which reasons between image and query for multiple rounds and reduces the referring ambiguity step by step. We show our new one-stage method obtains 5.0%, 4.5%, 7.5%, 12.8% absolute improvements over the state-of-the-art one-stage baseline on ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg, respectively. In particular, superior performances on longer and more complex queries validates the effectiveness of our query modeling.