ConFoThinking: Consolidated Focused Attention Driven Thinking for Visual Question Answering
This work improves the reliability of visual grounding and attention mechanisms for MLLMs in VQA, which is an incremental gain for researchers working on multimodal models.
This paper addresses the unreliability of grounding in MLLMs and fragmented attention signals in attention-driven methods for Visual Question Answering (VQA). The authors propose ConFoThinking, a framework that consolidates attention into a single intermediate layer and uses concise semantic cues for attention extraction, leading to significant improvements in perception performance across five VQA benchmarks.
Thinking with Images improves fine-grained VQA for MLLMs by emphasizing visual cues. However, tool-augmented methods depend on the capacity of grounding, which remains unreliable for MLLMs. In parallel, attention-driven methods to crop the Region of Interest (ROIs) are proposed but they are constrained by (1) fragmented attention signals scattered across layers, leading to suboptimal localization and (2) relying on question- or redundant-text-conditioned attention extraction. Our analysis reveals three patterns: MLLMs may attend to the correct region yet generate incorrect coordinates, where-to-look attention is often fragmented across layers, and attention extraction is query-sensitive. Motivated by these, We propose ConFoThinking, a Consolidated-Focused-Attention-Driven Thinking framework that learns to aggregate attention into a designated intermediate layer, from which we mine and zoom in salient regions for downstream visual understanding. Moreover, we extract attention using concise semantic cues of what to look into, which mitigates the semantic noise introduced by question- or redundant-text-based attention extraction. Experiments across five VQA benchmarks demonstrate ConFoThinking significantly improves perception performance. The code, checkpoints, and dataset will be released after being accepted.