Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation
This addresses the need for precise context retrieval in RAG systems, though it is incremental as it builds on existing methods.
The paper tackles the problem of retrieving specific regions within documents for retrieval-augmented generation by proposing a hybrid architecture that combines vision-language models' patch-level similarity with OCR-extracted bounding boxes, achieving improved precision without additional training.
Vision-language models (VLMs) like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they return entire pages rather than specific regions, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on retrieval precision. Our approach operates at inference time without additional training. We release Snappy, an open-source implementation demonstrating practical applicability, with empirical evaluation ongoing.