ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding
This addresses the challenge of limited multilingual captions for vision-and-language models, enabling broader multilingual capabilities, though it is an incremental approach by leveraging existing models.
The paper tackles the problem of language barriers in multilingual vision-and-language modeling by proposing ICU, which divides tasks into image captioning in English and cross-lingual language understanding, achieving new state-of-the-art results for five out of nine languages in the IGLUE benchmark.
Most multilingual vision-and-language (V&L) research aims to accomplish multilingual and multimodal capabilities within one model. However, the scarcity of multilingual captions for images has hindered the development. To overcome this obstacle, we propose ICU, Image Caption Understanding, which divides a V&L task into two stages: a V&L model performs image captioning in English, and a multilingual language model (mLM), in turn, takes the caption as the alt text and performs cross-lingual language understanding. The burden of multilingual processing is lifted off V&L model and placed on mLM. Since the multilingual text data is relatively of higher abundance and quality, ICU can facilitate the conquering of language barriers for V&L models. In experiments on two tasks across 9 languages in the IGLUE benchmark, we show that ICU can achieve new state-of-the-art results for five languages, and comparable results for the rest.