PRIM: Towards Practical In-Image Multilingual Machine Translation
This work addresses the lack of practical datasets for in-image multilingual machine translation, which is incremental as it builds on existing IIMT research by introducing real-world data.
The authors tackled the problem of in-image machine translation (IIMT) by addressing the gap between synthetic datasets and real-world conditions, resulting in the creation of the PRIM dataset and a model called VisTrans that improves translation quality and visual effects.
In-Image Machine Translation (IIMT) aims to translate images containing texts from one language to another. Current research of end-to-end IIMT mainly conducts on synthetic data, with simple background, single font, fixed text position, and bilingual translation, which can not fully reflect real world, causing a significant gap between the research and practical conditions. To facilitate research of IIMT in real-world scenarios, we explore Practical In-Image Multilingual Machine Translation (IIMMT). In order to convince the lack of publicly available data, we annotate the PRIM dataset, which contains real-world captured one-line text images with complex background, various fonts, diverse text positions, and supports multilingual translation directions. We propose an end-to-end model VisTrans to handle the challenge of practical conditions in PRIM, which processes visual text and background information in the image separately, ensuring the capability of multilingual translation while improving the visual quality. Experimental results indicate the VisTrans achieves a better translation quality and visual effect compared to other models. The code and dataset are available at: https://github.com/BITHLP/PRIM.