Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
This addresses the challenge of maintaining monolingual abilities like OCR while enhancing cross-lingual translation in MLLMs, which is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of catastrophic forgetting in Multimodal Large Language Models (MLLMs) when fine-tuned for Document Image Machine Translation (DIMT), by introducing a Synchronously Self-Reviewing (SSR) paradigm that prompts OCR text generation before translation, resulting in improved generalization on both OCR and DIMT tasks.
Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.