VAR: Visual Attention Reasoning via Structured Search and Backtracking
This addresses reliability issues in MLLMs for complex multimodal tasks, representing a novel method rather than an incremental improvement.
The paper tackles the problem of high hallucination and brittle reasoning in Multimodal Large Language Models (MLLMs) by introducing the Visual Attention Reasoning (VAR) framework, which achieves state-of-the-art results on hallucination and safety benchmarks, with a 7B model outperforming open-source models and competing with proprietary systems.
Multimodal Large Language Models (MLLMs), despite their advances, are hindered by their high hallucination tendency and heavy reliance on brittle, linear reasoning processes, leading to failures in complex tasks. To address these limitations, we introduce Visual Attention Reasoning (VAR), a novel framework that recasts grounded reasoning as a structured search over a reasoning trajectory space. VAR decomposes the reasoning process into two key stages: traceable evidence grounding and search-based chain-of-thought (CoT) generation, which incorporates a backtracking mechanism for self-correction. The search is guided by a multi-faceted reward function with semantic and geometric self-verification components, which penalize outputs that are not faithfully grounded in the visual input. We provide a theoretical analysis for our search strategy, validating its capability to find the correct solution with high probability. Experimental results show that our 7B model, VAR-7B, sets a new state-of-the-art on a comprehensive suite of hallucination and safety benchmarks, significantly outperforming existing open-source models and demonstrating competitive performance against leading proprietary systems.