Incorporating Visual Semantics into Sentence Representations within a Grounded Space
This work addresses the challenge of grounding language in vision for improved sentence representations, with incremental improvements in specific tasks.
The paper tackled the problem of learning sentence representations by incorporating visual semantics, overcoming the assumption of one-to-one correspondence between modalities. The proposed model outperformed the previous state-of-the-art on classification and semantic relatedness tasks.
Language grounding is an active field aiming at enriching textual representations with visual information. Generally, textual and visual elements are embedded in the same representation space, which implicitly assumes a one-to-one correspondence between modalities. This hypothesis does not hold when representing words, and becomes problematic when used to learn sentence representations --- the focus of this paper --- as a visual scene can be described by a wide variety of sentences. To overcome this limitation, we propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space. We further propose two new complementary objectives ensuring that (1) sentences associated with the same visual content are close in the grounded space and (2) similarities between related elements are preserved across modalities. We show that this model outperforms the previous state-of-the-art on classification and semantic relatedness tasks.