Who are you referring to? Coreference resolution in image narrations
This work addresses the problem of resolving coreferences in visual scene narrations for natural language processing and computer vision applications, representing an incremental advancement by extending an existing task to a new domain.
The paper tackles coreference resolution in long-form image narrations by introducing a new dataset with annotated coreference chains and bounding boxes, and proposes a weakly supervised technique that yields large performance gains over baselines.
Coreference resolution aims to identify words and phrases which refer to same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coreference chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improving grounding narratives in images.