Lessons learned in multilingual grounded language learning
This work addresses improving visual-semantic embeddings for multilingual applications, but it is incremental as it builds on existing methods by analyzing specific conditions.
The study investigated conditions affecting multilingual grounded language learning models, finding that multilingual training outperforms bilingual training, low-resource languages benefit from high-resource languages, and using translations or comparable pairs works equally well, with additional improvements from a caption-caption ranking objective.
Recent work has shown how to learn better visual-semantic embeddings by leveraging image descriptions in more than one language. Here, we investigate in detail which conditions affect the performance of this type of grounded language learning model. We show that multilingual training improves over bilingual training, and that low-resource languages benefit from training with higher-resource languages. We demonstrate that a multilingual model can be trained equally well on either translations or comparable sentence pairs, and that annotating the same set of images in multiple language enables further improvements via an additional caption-caption ranking objective.