MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG
For KB-VQA tasks requiring fine-grained visual discrimination, this work provides a novel agentic framework that reduces error propagation from retrieval, but the improvements are incremental over existing mRAG methods.
MMAgent-R² addresses the problem of visually similar distractors in KB-VQA by integrating visual reranking and active rejection into an agentic mRAG framework, achieving state-of-the-art performance on InfoSeek, E-VQA, and MMhops, with notable gains in challenging retrieval and multi-hop reasoning tasks.
Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions. Existing multimodal Retrieval Augmented Generation (mRAG) methods rely on global visual features to match candidate entities, yet when the knowledge base contains numerous visually similar entities, the retriever struggles to distinguish them, populating the candidate set with visually similar but factually mismatched distractors. Since subsequent processing steps such as noise filtering are also confined to this fixed candidate set, errors from failed retrieval inevitably propagate to the final answer. To address these challenges, we propose MMAgent-R$^2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism. Visual reranking directly compares query and candidate images, capturing discriminative details beyond textual descriptions to precisely identify the target entity among similar candidates; active rejection discards unreliable results and retrieves additional candidates when no confident match is found, moving beyond the fixed candidate pool. We design a composite reward function with step-level verification rewards and achieve joint optimization of external retrieval, internal verification, and answer generation via GRPO training. Experiments on InfoSeek, E-VQA, and MMhops demonstrate that \ours{} achieves state-of-the-art performance, with particularly notable advantages in challenging retrieval scenarios and complex multi-image multi-hop reasoning tasks.