An Effective Data Augmentation Method by Asking Questions about Scene Text Images
This work addresses accuracy issues in OCR for scene and handwritten text, offering an incremental improvement through structured auxiliary tasks.
The paper tackles the challenge of improving scene and handwritten text recognition by proposing a VQA-inspired data augmentation method that uses question-answering tasks to probe character-level attributes, resulting in consistent reductions in CER and WER on WordArt and Esposalles datasets.
Scene text recognition (STR) and handwritten text recognition (HTR) face significant challenges in accurately transcribing textual content from images into machine-readable formats. Conventional OCR models often predict transcriptions directly, which limits detailed reasoning about text structure. We propose a VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks. For each image-text pair, we generate natural-language questions probing character-level attributes such as presence, position, and frequency, with answers derived from ground-truth text. These auxiliary tasks encourage finer-grained reasoning, and the OCR model aligns visual features with textual queries to jointly reason over images and questions. Experiments on WordArt and Esposalles datasets show consistent improvements over baseline models, with significant reductions in both CER and WER. Our code is publicly available at https://github.com/xuyaooo/DataAugOCR.