Attention Grounded Enhancement for Visual Document Retrieval
This work addresses the challenge of accurate and interpretable retrieval for users needing to find documents based on implicit semantic connections, though it is incremental as it builds on existing fine-grained late interaction methods.
The paper tackles the problem of visual document retrieval by addressing the reliance on coarse global relevance labels, which leads to surface-level matching and difficulty with non-extractive queries, and proposes the AGREE framework that uses cross-modal attention as proxy local supervision to improve retrieval performance, achieving significant gains on the ViDoRe V2 benchmark.
Visual document retrieval requires understanding heterogeneous and multi-modal content to satisfy information needs. Recent advances use screenshot-based document encoding with fine-grained late interaction, significantly improving retrieval performance. However, retrievers are still trained with coarse global relevance labels, without revealing which regions support the match. As a result, retrievers tend to rely on surface-level cues and struggle to capture implicit semantic connections, hindering their ability to handle non-extractive queries. To alleviate this problem, we propose a \textbf{A}ttention-\textbf{G}rounded \textbf{RE}triever \textbf{E}nhancement (AGREE) framework. AGREE leverages cross-modal attention from multimodal large language models as proxy local supervision to guide the identification of relevant document regions. During training, AGREE combines local signals with the global signals to jointly optimize the retriever, enabling it to learn not only whether documents match, but also which content drives relevance. Experiments on the challenging ViDoRe V2 benchmark show that AGREE significantly outperforms the global-supervision-only baseline. Quantitative and qualitative analyses further demonstrate that AGREE promotes deeper alignment between query terms and document regions, moving beyond surface-level matching toward more accurate and interpretable retrieval. Our code is available at: https://anonymous.4open.science/r/AGREE-2025.