Multimodal Machine Translation with Visual Scene Graph Pruning
This work addresses a key bottleneck in multimodal machine translation for researchers and practitioners by proposing a novel method to reduce visual noise, though it appears incremental as it builds on existing scene graph and pruning techniques.
The paper tackles the problem of visual information redundancy in multimodal machine translation by introducing a visual Scene Graph Pruning (PSG) approach that uses language scene graphs to prune redundant nodes, reducing noise and improving translation performance, as demonstrated through comparative experiments with state-of-the-art methods.
Multimodal machine translation (MMT) seeks to address the challenges posed by linguistic polysemy and ambiguity in translation tasks by incorporating visual information. A key bottleneck in current MMT research is the effective utilization of visual data. Previous approaches have focused on extracting global or region-level image features and using attention or gating mechanisms for multimodal information fusion. However, these methods have not adequately tackled the issue of visual information redundancy in MMT, nor have they proposed effective solutions. In this paper, we introduce a novel approach--multimodal machine translation with visual Scene Graph Pruning (PSG), which leverages language scene graph information to guide the pruning of redundant nodes in visual scene graphs, thereby reducing noise in downstream translation tasks. Through extensive comparative experiments with state-of-the-art methods and ablation studies, we demonstrate the effectiveness of the PSG model. Our results also highlight the promising potential of visual information pruning in advancing the field of MMT.