Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation
This work addresses data scarcity and realism issues in multimodal machine translation, particularly for e-commerce applications, though it is incremental in leveraging existing data types.
The paper tackles the scarcity of triple data in multimodal machine translation by proposing a framework that leverages large-scale non-triple data, such as monolingual image-text and parallel text-only data, and introduces a new English-Chinese e-commerce dataset (EMMT) with ambiguous test cases. Experiments show significant translation performance improvements and competitiveness with state-of-the-art models on conventional benchmarks.
Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision. Previous MMT systems mainly focus on better access and use of visual information and tend to validate their methods on image-related datasets. These studies face two challenges. First, they can only utilize triple data (bilingual texts with images), which is scarce; second, current benchmarks are relatively restricted and do not correspond to realistic scenarios. Therefore, this paper correspondingly establishes new methods and new datasets for MMT. First, we propose a framework 2/3-Triplet with two new approaches to enhance MMT by utilizing large-scale non-triple data: monolingual image-text data and parallel text-only data. Second, we construct an English-Chinese {e}-commercial {m}ulti{m}odal {t}ranslation dataset (including training and testing), named EMMT, where its test set is carefully selected as some words are ambiguous and shall be translated mistakenly without the help of images. Experiments show that our method is more suitable for real-world scenarios and can significantly improve translation performance by using more non-triple data. In addition, our model also rivals various SOTA models in conventional multimodal translation benchmarks.