Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification
This addresses context-dependent VQA for AI systems, offering a novel approach to handle ambiguity, though it is incremental in combining reinforcement learning with existing VQA methods.
The paper tackles the problem of visual question answering (VQA) when questions are under-specified and require external context, proposing an agent that decides whether to ask for clarification or answer directly, which improves end-to-end VQA accuracy by an average of +15.3 points over baselines.
Real-world visual question answering (VQA) is often context-dependent: an image-question pair may be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose CoA(Clarify-or-Answer), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it asks a single focused question and then incorporates the response to produce the final answer. We introduce CONTEXTCLARIFY with a set of ambiguous VQA questions and the contrast set that is non-ambiguous. We further introduce GRPO-CR (Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, improving end-to-end VQA accuracy by an average of +15.3 points (83%) over prompting-based baselines