Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAG
It addresses a key bottleneck in multimodal AI for tasks requiring detailed image analysis, representing an incremental advance by adapting existing RAG techniques to a new domain.
This paper tackles the challenge of high-resolution image perception in multimodal large language models by introducing Retrieval-Augmented Perception (RAP), a training-free framework that uses retrieval-augmented generation to enhance long-context capabilities, resulting in LLaVA-v1.5-13B achieving a 43% improvement on V* Bench and 19% on HR-Bench.
High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs). To overcome the limitations of existing methods, this paper shifts away from prior dedicated heuristic approaches and revisits the most fundamental idea to HR perception by enhancing the long-context capability of MLLMs, driven by recent advances in long-context techniques like retrieval-augmented generation (RAG) for general LLMs. Towards this end, this paper presents the first study exploring the use of RAG to address HR perception challenges. Specifically, we propose Retrieval-Augmented Perception (RAP), a training-free framework that retrieves and fuses relevant image crops while preserving spatial context using the proposed Spatial-Awareness Layout. To accommodate different tasks, the proposed Retrieved-Exploration Search (RE-Search) dynamically selects the optimal number of crops based on model confidence and retrieval scores. Experimental results on HR benchmarks demonstrate the significant effectiveness of RAP, with LLaVA-v1.5-13B achieving a 43% improvement on $V^*$ Bench and 19% on HR-Bench.