Mixed Text Recognition with Efficient Parameter Fine-Tuning and Transformer
This addresses the problem of improving generality and stability in OCR systems for mixed text scenes, though it is incremental as it builds on pre-trained models with parameter-efficient fine-tuning.
The paper tackled the challenge of mixed-scene text recognition by proposing DLoRA-TrOCR, a parameter-efficient fine-tuning method that uses only 0.7% trainable parameters, achieving state-of-the-art results such as a CER of 4.02 on IAM and a WAR of 86.70 on the STR Benchmark.
With the rapid development of OCR technology, mixed-scene text recognition has become a key technical challenge. Although deep learning models have achieved significant results in specific scenarios, their generality and stability still need improvement, and the high demand for computing resources affects flexibility. To address these issues, this paper proposes DLoRA-TrOCR, a parameter-efficient hybrid text spotting method based on a pre-trained OCR Transformer. By embedding a weight-decomposed DoRA module in the image encoder and a LoRA module in the text decoder, this method can be efficiently fine-tuned on various downstream tasks. Our method requires no more than 0.7\% trainable parameters, not only accelerating the training efficiency but also significantly improving the recognition accuracy and cross-dataset generalization performance of the OCR system in mixed text scenes. Experiments show that our proposed DLoRA-TrOCR outperforms other parameter-efficient fine-tuning methods in recognizing complex scenes with mixed handwritten, printed, and street text, achieving a CER of 4.02 on the IAM dataset, a F1 score of 94.29 on the SROIE dataset, and a WAR of 86.70 on the STR Benchmark, reaching state-of-the-art performance.