Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding
This addresses efficiency and accuracy challenges for LVLMs in handling complex visual data, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the computational inefficiency of Large Vision-Language Models (LVLMs) when processing information-rich images like infographics by proposing PinPoint, a two-stage framework that identifies instruction-relevant regions to reduce visual tokens, achieving superior accuracy and reducing computational overhead.
Large Vision-Language Models (LVLMs) have shown strong performance across various multimodal tasks by leveraging the reasoning capabilities of Large Language Models (LLMs). However, processing visually complex and information-rich images, such as infographics or document layouts, requires these models to generate a large number of visual tokens, leading to significant computational overhead. To address this, we propose PinPoint, a novel two-stage framework that first identifies instruction-relevant image regions and then refines them to extract fine-grained visual features for improved reasoning and efficiency. Central to our approach is the Instruction-Region Alignment, which localizes relevant regions using both visual input and textual instructions. We further introduce new annotations that provide richer ground-truth supervision for instruction-relevant regions across challenging VQA benchmarks: InfographicVQA, MultiPageDocVQA, and SinglePageDocVQA. Experimental results show that PinPoint not only achieves superior accuracy compared to existing methods but also reduces computational overhead by minimizing irrelevant visual tokens.