DRISHTIKON: Visual Grounding at Multiple Granularities in Documents
This addresses the problem of interpretability and trust in Visual Question Answering for complex, multilingual documents, though it appears incremental as it builds on existing components like OCR and LLMs with a novel region matching algorithm.
The paper tackles visual grounding in text-rich document images by introducing DRISHTIKON, a framework that localizes answer spans at multiple granularities (block, line, word, point), achieving state-of-the-art grounding accuracy with line-level granularity providing the best balance between precision and recall.
Visual grounding in text-rich document images is a critical yet underexplored challenge for Document Intelligence and Visual Question Answering (VQA) systems. We present DRISHTIKON, a multi-granular and multi-block visual grounding framework designed to enhance interpretability and trust in VQA for complex, multilingual documents. Our approach integrates multilingual OCR, large language models, and a novel region matching algorithm to localize answer spans at the block, line, word, and point levels. We introduce the Multi-Granular Visual Grounding (MGVG) benchmark, a curated test set of diverse circular notifications from various sectors, each manually annotated with fine-grained, human-verified labels across multiple granularities. Extensive experiments show that our method achieves state-of-the-art grounding accuracy, with line-level granularity providing the best balance between precision and recall. Ablation studies further highlight the benefits of multi-block and multi-line reasoning. Comparative evaluations reveal that leading vision-language models struggle with precise localization, underscoring the effectiveness of our structured, alignment-based approach. Our findings pave the way for more robust and interpretable document understanding systems in real-world, text-centric scenarios with multi-granular grounding support. Code and dataset are made available for future research.