Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction
This addresses the challenge of human-robot interaction by allowing more natural language commands, though it appears incremental as it builds on existing grounding methods.
The paper tackles the problem of enabling robots to retrieve everyday objects using unconstrained natural language descriptions by developing a two-stage neural-network pipeline for semantic and spatial grounding, and preliminary results show it outperforms a near state-of-the-art system on standard benchmarks.
The human language is one of the most natural interfaces for humans to interact with robots. This paper presents a robot system that retrieves everyday objects with unconstrained natural language descriptions. A core issue for the system is semantic and spatial grounding, which is to infer objects and their spatial relationships from images and natural language expressions. We introduce a two-stage neural-network grounding pipeline that maps natural language referring expressions directly to objects in the images. The first stage uses visual descriptions in the referring expressions to generate a candidate set of relevant objects. The second stage examines all pairwise relationships between the candidates and predicts the most likely referred object according to the spatial descriptions in the referring expressions. A key feature of our system is that by leveraging a large dataset of images labeled with text descriptions, it allows unrestricted object types and natural language referring expressions. Preliminary results indicate that our system outperforms a near state-of-the-art object comprehension system on standard benchmark datasets. We also present a robot system that follows voice commands to pick and place previously unseen objects.