Encoding Spatial Relations from Natural Language
This addresses a limitation in NLP for applications requiring spatial reasoning, such as robotics or scene understanding, but is incremental as it builds on existing multi-modal approaches.
The paper tackles the problem of learning spatial relations from natural language, which existing methods fail to capture robustly, by introducing a novel multi-modal objective and dataset, resulting in representations that achieve paraphrase and viewpoint invariance.
Natural language processing has made significant inroads into learning the semantics of words through distributional approaches, however representations learnt via these methods fail to capture certain kinds of information implicit in the real world. In particular, spatial relations are encoded in a way that is inconsistent with human spatial reasoning and lacking invariance to viewpoint changes. We present a system capable of capturing the semantics of spatial relations such as behind, left of, etc from natural language. Our key contributions are a novel multi-modal objective based on generating images of scenes from their textual descriptions, and a new dataset on which to train it. We demonstrate that internal representations are robust to meaning preserving transformations of descriptions (paraphrase invariance), while viewpoint invariance is an emergent property of the system.