ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
This addresses the challenge of noisy retrieval and limited reasoning in KB-VQA for AI systems handling knowledge-intensive multimodal tasks, representing an incremental improvement over existing methods.
The paper tackles the problem of knowledge-based visual question answering (KB-VQA) where multimodal large language models struggle with domain-specific queries, proposing ReAG, a reasoning-augmented approach that improves answer accuracy on benchmarks like Encyclopedic-VQA and InfoSeek.
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA). However, even state-of-the-art MLLMs struggle with domain-specific or knowledge-intensive queries, where relevant information is underrepresented in pre-training data. Knowledge-based VQA (KB-VQA) addresses this by retrieving external documents to condition answer generation, but current retrieval-augmented approaches suffer from low precision, noisy passages, and limited reasoning. To address this, we propose ReAG, a novel Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages, ensuring high-quality additional context. The model follows a multi-stage training strategy leveraging reinforcement learning to enhance reasoning over retrieved content, while supervised fine-tuning serves only as a cold start. Extensive experiments on Encyclopedic-VQA and InfoSeek demonstrate that ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence. Our source code is publicly available at: https://github.com/aimagelab/ReAG.