Decoupling Reasoning and Perception: An LLM-LMM Framework for Faithful Visual Reasoning
This addresses a critical limitation in visual reasoning for AI systems, though it is an incremental improvement as it builds on existing LLM and LMM paradigms without new training.
The paper tackles the problem of Large Multimodal Models (LMMs) losing grounding in visual information during extended reasoning, leading to errors, by introducing a training-free framework that decouples reasoning and perception using an LLM and LMM. The result is a significant reduction in visually-unfounded reasoning steps and substantial improvement in reasoning fidelity.
Significant advancements in the reasoning capabilities of Large Language Models (LLMs) are now driven by test-time scaling laws, particularly those leveraging extended Chain-of-Thought (CoT) reasoning. Inspired by these breakthroughs, researchers have extended these paradigms to Large Multimodal Models (LMMs). However, a critical limitation emerges: as their reasoning chains extend, LMMs increasingly rely on textual logic, progressively losing grounding in the underlying visual information. This leads to reasoning paths that diverge from the image content, culminating in erroneous conclusions. To address this, we introduce a strikingly simple yet effective training-free visual-reasoning pipeline. The core concept is to decouple the reasoning and perception processes. A powerful LLM orchestrates the high-level reasoning, strategically interrogating a LMM to extract specific visual information required for its logical chain. The LMM, in turn, functions exclusively as a visual question-answering engine, supplying the necessary perceptual details on demand. This lightweight, plug-and-play approach requires no additional training or architectural changes. Comprehensive evaluations validate that our framework effectively governs the visual reasoning process, leading to a significant reduction in visually-unfounded reasoning steps and a substantial improvement in reasoning fidelity.