Referring Expression Comprehension: A Survey of Methods and Datasets
It provides a comprehensive overview for researchers in computer vision and natural language processing, but it is incremental as it synthesizes existing work without introducing new methods or results.
This survey examines the state-of-the-art methods and datasets for referring expression comprehension (REC), a task that localizes objects in images based on natural language descriptions, and reviews approaches like CNN-RNN models, modular networks, and graph-based models while comparing results across datasets and settings.
Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It thus more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In this survey, we first examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to encode the visual and textual modalities. In particular, we examine the common approach of joint embedding images and expressions to a common feature space. We also discuss modular architectures and graph-based models that interface with structured graph representation. In the second part of this survey, we review the datasets available for training and evaluating REC systems. We then group results according to the datasets, backbone models, settings so that they can be fairly compared. Finally, we discuss promising future directions for the field, in particular the compositional referring expression comprehension that requires longer reasoning chain to address.