ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge
This provides a diagnostic tool for researchers and practitioners in AI and multimodal systems to assess and improve retrieval models on complex tasks, though it is incremental as it builds on existing benchmark concepts.
The authors tackled the lack of multimodal retrieval benchmarks for professional knowledge and complex reasoning by introducing ARK, a benchmark that evaluates retrieval across knowledge domains and reasoning skills, revealing gaps and bottlenecks in existing models.
Existing multimodal retrieval benchmarks largely emphasize semantic matching on daily-life images and offer limited diagnostics of professional knowledge and complex reasoning. To address this gap, we introduce ARK, a benchmark designed to analyze multimodal retrieval from two complementary perspectives: (i) knowledge domains (five domains with 17 subtypes), which characterize the content and expertise retrieval relies on, and (ii) reasoning skills (six categories), which characterize the type of inference over multimodal evidence required to identify the correct candidate. Specifically, ARK evaluates retrieval with both unimodal and multimodal queries and candidates, covering 16 heterogeneous visual data types. To avoid shortcut matching during evaluation, most queries are paired with targeted hard negatives that require multi-step reasoning. We evaluate 23 representative text-based and multimodal retrievers on ARK and observe a pronounced gap between knowledge-intensive and reasoning-intensive retrieval, with fine-grained visual and spatial reasoning emerging as persistent bottlenecks. We further show that simple enhancements such as re-ranking and rewriting yield consistent improvements, but substantial headroom remains.