MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios
This work addresses the lack of systematic evaluation of multimodal embedding models in long-context scenarios, which is critical for real-world deployment.
The paper introduces MMLongEmbed, the first benchmark for evaluating multimodal embedding models in long-context scenarios, and finds that current models rely on superficial feature matching and struggle with deep semantic dependencies, with performance degrading systematically with context length and key information placement.
Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context windows do not necessarily translate into effective comprehension and representation of long-context multimodal inputs, which remains a critical bottleneck for real-world deployment. To address the lack of systematic evaluation in this setting, we introduce MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios. MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities. Through extensive evaluation of state-of-the-art models, we find that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies. We further observe that performance degradation varies systematically with context length and key information placement. Moreover, models exhibit substantially different robustness to redundant contextual information across modalities. For reproducibility, the benchmark and code are publicly available.