Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation
This addresses a practical problem for multimodal machine translation by enabling the use of synthetic images without performance degradation, though it is incremental as it builds on existing methods.
The paper tackles the distribution shift between synthetic and authentic images in multimodal machine translation by minimizing the gap in image representations and decoder outputs, achieving state-of-the-art performance on Multi30K En-De and En-Fr datasets without needing authentic images during inference.
Multimodal machine translation (MMT) simultaneously takes the source sentence and a relevant image as input for translation. Since there is no paired image available for the input sentence in most cases, recent studies suggest utilizing powerful text-to-image generation models to provide image inputs. Nevertheless, synthetic images generated by these models often follow different distributions compared to authentic images. Consequently, using authentic images for training and synthetic images for inference can introduce a distribution shift, resulting in performance degradation during inference. To tackle this challenge, in this paper, we feed synthetic and authentic images to the MMT model, respectively. Then we minimize the gap between the synthetic and authentic images by drawing close the input image representations of the Transformer Encoder and the output distributions of the Transformer Decoder. Therefore, we mitigate the distribution disparity introduced by the synthetic images during inference, thereby freeing the authentic images from the inference process.Experimental results show that our approach achieves state-of-the-art performance on the Multi30K En-De and En-Fr datasets, while remaining independent of authentic images during inference.