Multilingual Multi-modal Embeddings for Natural Language Processing
This work addresses the challenge of leveraging diverse data sources for better embeddings in NLP, but it is incremental as it builds on existing contrastive methods.
The authors tackled the problem of learning embeddings from multilingual and multi-modal data, resulting in improvements on image-sentence ranking and semantic textual similarity tasks, with strong gains in neural machine translation re-ranking.
We propose a novel discriminative model that learns embeddings from multilingual and multi-modal data, meaning that our model can take advantage of images and descriptions in multiple languages to improve embedding quality. To that end, we introduce a modification of a pairwise contrastive estimation optimisation function as our training objective. We evaluate our embeddings on an image-sentence ranking (ISR), a semantic textual similarity (STS), and a neural machine translation (NMT) task. We find that the additional multilingual signals lead to improvements on both the ISR and STS tasks, and the discriminative cost can also be used in re-ranking $n$-best lists produced by NMT models, yielding strong improvements.