Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs
This addresses the problem of limited generalizability and increased complexity in fine-tuning MLLMs for tasks like visual search and video understanding, offering a simpler alternative.
The paper tackles MLLMs' difficulty with fine-grained perception in complex visual tasks by proposing a training-free framework that uses the model's intrinsic uncertainty to guide attention to salient visual information, achieving performance competitive with specialized fine-tuned methods.
Multimodal Large Language Models (MLLMs) often struggle with fine-grained perception, such as identifying small objects in high-resolution images or finding key moments in long videos. Existing works typically rely on complicated, task-specific fine-tuning, which limits their generalizability and increases model complexity. In this work, we propose an effective, training-free framework that uses an MLLM's intrinsic uncertainty as a proactive guidance signal. Our core insight is that a model's output entropy decreases when presented with relevant visual information. We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most salient data. We apply this simple principle to three complex visual tasks: Visual Search, Long Video Understanding, and Temporal Grounding, allowing off-the-shelf MLLMs to achieve performance competitive with specialized, fine-tuned methods. Our work validates that harnessing intrinsic uncertainty is a powerful, general strategy for enhancing fine-grained multimodal performance.