ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation
This addresses the problem of extending vision-language models to non-English and multimodal generation tasks for multilingual AI applications, representing a novel integration rather than an incremental improvement.
The paper tackles the limitation of existing cross-lingual cross-modal models that only handle understanding tasks by proposing ERNIE-UniX2, a unified framework for both generation and understanding tasks, achieving state-of-the-art results on tasks like multimodal machine translation and multilingual visual question answering.
Recent cross-lingual cross-modal works attempt to extend Vision-Language Pre-training (VLP) models to non-English inputs and achieve impressive performance. However, these models focus only on understanding tasks utilizing encoder-only architecture. In this paper, we propose ERNIE-UniX2, a unified cross-lingual cross-modal pre-training framework for both generation and understanding tasks. ERNIE-UniX2 integrates multiple pre-training paradigms (e.g., contrastive learning and language modeling) based on encoder-decoder architecture and attempts to learn a better joint representation across languages and modalities. Furthermore, ERNIE-UniX2 can be seamlessly fine-tuned for varieties of generation and understanding downstream tasks. Pre-trained on both multilingual text-only and image-text datasets, ERNIE-UniX2 achieves SOTA results on various cross-lingual cross-modal generation and understanding tasks such as multimodal machine translation and multilingual visual question answering.