Beyond Bag-of-Patches: Learning Global Layout via Textual Supervision for Late-Interaction Visual Document Retrieval
For practitioners of visual document retrieval, this work improves retrieval accuracy by incorporating global layout information without changing inference, addressing a known limitation of late interaction models.
Visual Document Retrieval models using late interaction architectures struggle with global layout understanding. The authors propose a multimodal encoder that augments local patch embeddings with a global layout embedding trained via textual descriptions, achieving +2.4 nDCG@5 and +2.3 MAP@5 over ColPali/ColQwen baselines on ViDoRe-v2 datasets.
Visual Document Retrieval (VDR) models mostly rely on late interaction architectures, in which documents are represented by a set of local patch embeddings and then matched against query tokens. While efficient, this architecture prioritizes local similarity over global layout structure of documents to estimate relevancy between documents and query. In practice, this leads to errors as relevance originates from layout structure of documents with heterogeneous layouts combining figures, tables, and text. We make document layout learnable without changing inference. We propose a multimodal encoder that augments local patch representations with a global layout embedding, trained via textual descriptions encoding document layout information. Across four ViDoRe-v2 datasets, our model improves over the strongest architecturally comparable ColPali/ColQwen baseline by +2.4 nDCG@5 and +2.3 MAP@5, with statistically significant per-dataset gains over ColQwen.