Look Twice: Training-Free Evidence Highlighting in Multimodal Large Language Models
This addresses the challenge of noisy or partially relevant evidence integration in multimodal question-answering for users of such models, though it is incremental as it builds on existing pretrained models without architectural changes.
The paper tackles the problem of multimodal large language models struggling to identify relevant visual and textual evidence for knowledge-intensive queries by introducing Look Twice (LoT), a training-free inference-time framework that highlights evidence using attention patterns and prompt markers, resulting in consistent improvements over zero-shot MLLMs across multiple benchmarks.
Answering questions about images often requires combining visual understanding with external knowledge. Multimodal Large Language Models (MLLMs) provide a natural framework for this setting, but they often struggle to identify the most relevant visual and textual evidence when answering knowledge-intensive queries. In such scenarios, models must integrate visual cues with retrieved textual evidence that is often noisy or only partially relevant, while also localizing fine-grained visual information in the image. In this work, we introduce Look Twice (LoT), a training-free inference-time framework that improves how pretrained MLLMs utilize multimodal evidence. Specifically, we exploit the model attention patterns to estimate which visual regions and retrieved textual elements are relevant to a query, and then generate the answer conditioned on this highlighted evidence. The selected cues are highlighted through lightweight prompt-level markers that encourage the model to re-attend to the relevant evidence during generation. Experiments across multiple knowledge-based VQA benchmarks show consistent improvements over zero-shot MLLMs. Additional evaluations on vision-centric and hallucination-oriented benchmarks further demonstrate that visual evidence highlighting alone improves model performance in settings without textual context, all without additional training or architectural modifications. Source code will be publicly released.