CMDR: Contextual Multimodal Document Retrieval
For researchers in multimodal retrieval and document understanding, this work addresses the overlooked problem of cross-page contextual queries, providing a new benchmark and method.
CMDR introduces a new multimodal document retrieval task and benchmark (CMDR-Bench) that requires modeling document context across pages, and proposes CMDR-Embed, a contextual multimodal embedding framework that jointly encodes multiple pages to improve retrieval. Experiments show CMDR-Embed significantly outperforms non-contextual embeddings.
Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing benchmarks primarily evaluate simple lexical or semantic matching, and most methods encode pages independently. Consequently, they overlook the contextual information in the document required to resolve queries that aggregate information across multiple pages. In this paper, we introduce CMDR and CMDR-Bench, a new multimodal document retrieval task and benchmark that require modeling document context. To address this challenge, we propose CMDR-Embed, a contextual multimodal embedding framework that explicitly incorporates document context by jointly encoding multiple pages and deriving page-level embeddings from a shared contextual representation. Furthermore, we introduce CMCL, a contextual multimodal contrastive learning objective that effectively trains CMDR-Embed by balancing contextual modeling with page-level discriminability. Experiments demonstrate that CMDR-Embed significantly outperforms non-contextual embeddings, highlighting the importance of context-aware multimodal embeddings for advancing document retrieval.