Transferring General Multimodal Pretrained Models to Text Recognition
This addresses text recognition, particularly for Chinese, by enabling efficient transfer from general models, though it is incremental as it builds on existing multimodal frameworks.
The paper tackles text recognition by reformulating it as an image captioning task and transferring a multimodal pretrained model, achieving state-of-the-art performance on a Chinese benchmark without large-scale text-specific pretraining.
This paper proposes a new method, OFA-OCR, to transfer multimodal pretrained models to text recognition. Specifically, we recast text recognition as image captioning and directly transfer a unified vision-language pretrained model to the end task. Without pretraining on large-scale annotated or synthetic text recognition data, OFA-OCR outperforms the baselines and achieves state-of-the-art performance in the Chinese text recognition benchmark. Additionally, we construct an OCR pipeline with OFA-OCR, and we demonstrate that it can achieve competitive performance with the product-level API. The code (https://github.com/OFA-Sys/OFA) and demo (https://modelscope.cn/studios/damo/ofa_ocr_pipeline/summary) are publicly available.