Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering
This addresses overfitting issues in ST-VQA for applications requiring robust text understanding in images, though it is incremental as it builds on existing adversarial training methods.
The paper tackles the problem of overfitting in Scene-Text Visual Question Answering (ST-VQA) due to reliance on noisy OCR data by proposing a multimodal adversarial training architecture with an Adversarial OCR Enhancement module and Spatial-Aware Self-Attention, achieving significant performance improvements on ST-VQA and TextVQA datasets.
Scene-Text Visual Question Answering (ST-VQA) aims to understand scene text in images and answer questions related to the text content. Most existing methods heavily rely on the accuracy of Optical Character Recognition (OCR) systems, and aggressive fine-tuning based on limited spatial location information and erroneous OCR text information often leads to inevitable overfitting. In this paper, we propose a multimodal adversarial training architecture with spatial awareness capabilities. Specifically, we introduce an Adversarial OCR Enhancement (AOE) module, which leverages adversarial training in the embedding space of OCR modality to enhance fault-tolerant representation of OCR texts, thereby reducing noise caused by OCR errors. Simultaneously, We add a Spatial-Aware Self-Attention (SASA) mechanism to help the model better capture the spatial relationships among OCR tokens. Various experiments demonstrate that our method achieves significant performance improvements on both the ST-VQA and TextVQA datasets and provides a novel paradigm for multimodal adversarial training.