An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
This addresses the challenge of enhancing translation accuracy for multimodal tasks, though it appears incremental as it builds on existing attention-based methods.
The paper tackled the problem of improving Neural Machine Translation (NMT) by incorporating images through multimodal attention mechanisms, achieving state-of-the-art scores on the Multi30k dataset.
In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several attention mechanism on the multimodal translation task (English, image to German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless identify and report different misbehavior of the machine while translating.