Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
This work addresses the problem of improving translation accuracy in unsupervised multimodal settings for researchers and practitioners, representing an incremental advancement by integrating visual cues into existing unsupervised methods.
The paper tackles the challenge of aligning source-target sentences in unsupervised multimodal machine translation by leveraging visual content for disambiguation and latent space alignment, resulting in significant improvements over state-of-the-art methods on the Multi30K dataset with good generalization when images are unavailable at test time.
Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when the images are not available at the testing time.