GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation
This work addresses the problem of modality gaps and domain limitations in multimodal machine translation for researchers and practitioners in natural language processing.
The paper tackles the challenge of multimodal machine translation (MMT) by proposing GIIFT, a framework that uses multimodal scene graphs and a cross-modal Graph Attention Network adapter to learn multimodal knowledge and generalize it to image-free translation domains. Experimental results on Multi30K and WMT benchmarks show that GIIFT achieves state-of-the-art performance in English-to-French and English-to-German tasks, even without images during inference.
Multimodal Machine Translation (MMT) has demonstrated the significant help of visual information in machine translation. However, existing MMT methods face challenges in leveraging the modality gap by enforcing rigid visual-linguistic alignment whilst being confined to inference within their trained multimodal domains. In this work, we construct novel multimodal scene graphs to preserve and integrate modality-specific information and introduce GIIFT, a two-stage Graph-guided Inductive Image-Free MMT framework that uses a cross-modal Graph Attention Network adapter to learn multimodal knowledge in a unified fused space and inductively generalize it to broader image-free translation domains. Experimental results on the Multi30K dataset of English-to-French and English-to-German tasks demonstrate that our GIIFT surpasses existing approaches and achieves the state-of-the-art, even without images during inference. Results on the WMT benchmark show significant improvements over the image-free translation baselines, demonstrating the strength of GIIFT towards inductive image-free inference.