Panoptic Narrative Grounding
This work addresses the challenge of fine-grained visual grounding for computer vision researchers, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of natural language visual grounding by proposing a spatially fine and general formulation called Panoptic Narrative Grounding, establishing an experimental framework with new ground truth and metrics, and achieving a baseline performance of 55.4 absolute Average Recall points.
This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, including new ground truth and metrics, and we propose a strong baseline method to serve as stepping stone for future work. We exploit the intrinsic semantic richness in an image by including panoptic categories, and we approach visual grounding at a fine-grained level by using segmentations. In terms of ground truth, we propose an algorithm to automatically transfer Localized Narratives annotations to specific regions in the panoptic segmentations of the MS COCO dataset. To guarantee the quality of our annotations, we take advantage of the semantic structure contained in WordNet to exclusively incorporate noun phrases that are grounded to a meaningfully related panoptic segmentation region. The proposed baseline achieves a performance of 55.4 absolute Average Recall points. This result is a suitable foundation to push the envelope further in the development of methods for Panoptic Narrative Grounding.