Give Me Something to Eat: Referring Expression Comprehension with Commonsense Knowledge
This addresses a practical limitation in AI for human-computer interaction where users query objects based on affordances rather than visual cues, though it is incremental as it builds on existing REF frameworks.
The paper tackles the problem of referring expression comprehension when queries involve commonsense knowledge, such as 'Give me something to eat', by introducing a new dataset KB-Ref with 43k expressions on 16k images and showing that state-of-the-art models suffer a large performance drop on it. Their proposed ECIFA network significantly improves over these models, though a gap remains compared to human performance.
Conventional referring expression comprehension (REF) assumes people to query something from an image by describing its visual appearance and spatial location, but in practice, we often ask for an object by describing its affordance or other non-visual attributes, especially when we do not have a precise target. For example, sometimes we say 'Give me something to eat'. In this case, we need to use commonsense knowledge to identify the objects in the image. Unfortunately, these is no existing referring expression dataset reflecting this requirement, not to mention a model to tackle this challenge. In this paper, we collect a new referring expression dataset, called KB-Ref, containing 43k expressions on 16k images. In KB-Ref, to answer each expression (detect the target object referred by the expression), at least one piece of commonsense knowledge must be required. We then test state-of-the-art (SoTA) REF models on KB-Ref, finding that all of them present a large drop compared to their outstanding performance on general REF datasets. We also present an expression conditioned image and fact attention (ECIFA) network that extract information from correlated image regions and commonsense knowledge facts. Our method leads to a significant improvement over SoTA REF models, although there is still a gap between this strong baseline and human performance. The dataset and baseline models will be released.