Towards High-Resolution Visual Perception via Hierarchical Entity Exploration
For practitioners using multimodal LLMs on high-resolution images, this work offers a training-free method to improve fine-grained visual perception without requiring model retraining.
The paper introduces Hierarchical Entity Exploration (HEE), a training-free framework for high-resolution image perception in MLLMs that dynamically explores image regions guided by queries. HEE outperforms existing training-free methods like ZoomEye and RAP on Visual Probe and HR-Bench benchmarks across multiple MLLMs, achieving higher accuracy and efficiency.
High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs), as fine-grained details are often lost when the image is processed as a whole. Existing methods either require training to teach models where to look or heuristically divide the image into fixed regions, both of which struggle to generalize in complex HR scenes. In this work, we propose Hierarchical Entity Exploration (HEE), a training-free and model-agnostic framework that transforms static image understanding into dynamic, query-guided entity exploration. HEE first evaluates each region using a dual scoring mechanism to determine whether it already contains sufficient evidence to answer the question. If not, it applies object detection within the most promising region to extract fine-grained entities, clusters them into coherent subregions, and organizes them into a multi-level semantic hierarchy for deeper exploration. When deeper regions still fail to yield confident answers, a confidence-guided backtracking mechanism revisits alternative paths to ensure adaptive perception. Extensive results show that HEE outperforms training-free methods like ZoomEye and RAP in both accuracy and efficiency on two complex HR benchmarks (Visual Probe and HR-Bench), across different MLLMs such as Qwen2.5-VL and LLaVA-OneVision. Moreover, HEE demonstrates generalization on the MME-RealWorld benchmark.