ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval
This addresses the need for better retrieval-augmented generation in AI systems requiring specialized or up-to-date knowledge from multimodal documents, though it appears incremental as it builds on existing retrieval techniques.
The paper tackled the problem of multimodal document retrieval by introducing ColMate, a model that uses OCR-based pretraining, masked contrastive learning, and late interaction scoring, achieving a 3.61% improvement over existing models on the ViDoRe V2 benchmark.
Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks.