Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs
This addresses the issue of underutilized visual information in MMT for researchers and practitioners, representing an incremental improvement through a novel hybrid method.
This paper tackles the problem of multimodal machine translation (MMT) systems showing decreased sensitivity to visual information by attributing it to insufficient cross-modal interaction and proposing a novel approach using Visual Question-Answering (VQA) pairs generated via Large Language Models to enhance interaction. The method, tested on two benchmarks, demonstrates effectiveness, with code and data made available.
This paper presents an in-depth study of multimodal machine translation (MMT), examining the prevailing understanding that MMT systems exhibit decreased sensitivity to visual information when text inputs are complete. Instead, we attribute this phenomenon to insufficient cross-modal interaction, rather than image information redundancy. A novel approach is proposed to generate parallel Visual Question-Answering (VQA) style pairs from the source text, fostering more robust cross-modal interaction. Using Large Language Models (LLMs), we explicitly model the probing signal in MMT to convert it into VQA-style data to create the Multi30K-VQA dataset. An MMT-VQA multitask learning framework is introduced to incorporate explicit probing signals from the dataset into the MMT training process. Experimental results on two widely-used benchmarks demonstrate the effectiveness of this novel approach. Our code and data would be available at: \url{https://github.com/libeineu/MMT-VQA}.