Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling
This work addresses scene-text understanding for multimodal AI systems, representing an incremental improvement in Text-VQA.
The paper tackles the Text-VQA problem by proposing the LOGOS model, which localizes key image information, groups OCR tokens, and selects answers from OCR texts, achieving state-of-the-art performance on two benchmarks without extra OCR annotations.
As an important task in multimodal context understanding, Text-VQA (Visual Question Answering) aims at question answering through reading text information in images. It differentiates from the original VQA task as Text-VQA requires large amounts of scene-text relationship understanding, in addition to the cross-modal grounding capability. In this paper, we propose Localize, Group, and Select (LOGOS), a novel model which attempts to tackle this problem from multiple aspects. LOGOS leverages two grounding tasks to better localize the key information of the image, utilizes scene text clustering to group individual OCR tokens, and learns to select the best answer from different sources of OCR (Optical Character Recognition) texts. Experiments show that LOGOS outperforms previous state-of-the-art methods on two Text-VQA benchmarks without using additional OCR annotation data. Ablation studies and analysis demonstrate the capability of LOGOS to bridge different modalities and better understand scene text.