Generalization algorithm of multimodal pre-training model based on graph-text self-supervised training
This addresses the problem of visual information scarcity and inaccuracy for researchers and practitioners in multimodal machine translation, though it appears incremental as it builds on existing multimodal pre-training approaches.
The paper tackles the problem of limited visual information availability and effectiveness in multimodal neural machine translation by proposing a self-supervised training algorithm that retrieves images from existing sentences and uses graph-text relationships to generate more effective visual information. The result shows a 0.5 BLEU improvement over the baseline on the global voice dataset when using the filtered information for fine-tuning.
Recently, a large number of studies have shown that the introduction of visual information can effectively improve the effect of neural machine translation (NMT). Its effectiveness largely depends on the availability of a large number of bilingual parallel sentence pairs and manual image annotation. The lack of images and the effectiveness of images have been difficult to solve. In this paper, a multimodal pre-training generalization algorithm for self-supervised training is proposed, which overcomes the lack of visual information and inaccuracy, and thus extends the applicability of images on NMT. Specifically, we will search for many pictures from the existing sentences through the search engine, and then through the relationship between visual information and text, do the self-supervised training task of graphics and text to obtain more effective visual information for text. We show that when the filtered information is used as multimodal machine translation for fine-tuning, the effect of translation in the global voice dataset is 0.5 BLEU higher than the baseline.