Revisiting Visual Grounding
This work highlights dataset biases and method limitations in visual grounding, which is important for researchers in language-vision integration, though it is incremental as it builds on prior analyses.
The study revisited the IRSG visual grounding method and its datasets, finding that the system does not effectively use learned object-relationship models and that the datasets contain biases allowing methods ignoring relationships to perform well, with experiments on a cleaned subset addressing these issues.
We revisit a particular visual grounding method: the "Image Retrieval Using Scene Graphs" (IRSG) system of Johnson et al. (2015). Our experiments indicate that the system does not effectively use its learned object-relationship models. We also look closely at the IRSG dataset, as well as the widely used Visual Relationship Dataset (VRD) that is adapted from it. We find that these datasets exhibit biases that allow methods that ignore relationships to perform relatively well. We also describe several other problems with the IRSG dataset, and report on experiments using a subset of the dataset in which the biases and other problems are removed. Our studies contribute to a more general effort: that of better understanding what machine learning methods that combine language and vision actually learn and what popular datasets actually test.