MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
For researchers developing clinical AI, this benchmark exposes critical gaps in rare-disease multimodal reasoning, particularly the capacity dilution effect where medical fine-tuning improves diagnostics but harms multi-image integration.
MMRareBench is the first benchmark for evaluating multimodal large language models on rare diseases across four clinical tracks. Evaluation of 23 models reveals fragmented capabilities and universally low treatment-planning performance, with medical-domain models lagging behind general-purpose models on multi-image tasks.
Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely untested. In rare-disease scenarios, clinicians often lack prior clinical knowledge, forcing them to rely strictly on case-level evidence for clinical judgments. Existing benchmarks predominantly evaluate common-condition, single-image settings, leaving multimodal and multi-image evidence integration under rare-disease data scarcity systematically unevaluated. We introduce MMRareBench, to our knowledge the first rare-disease benchmark jointly evaluating multimodal and multi-image clinical capability across four workflow-aligned tracks: diagnosis, treatment planning, cross-image evidence alignment, and examination suggestion. The benchmark comprises 1,756 question-answer pairs with 7,958 associated medical images curated from PMC case reports, with Orphanet-anchored ontology alignment, track-specific leakage control, evidence-grounded annotations, and a two-level evaluation protocol. A systematic evaluation of 23 MLLMs reveals fragmented capability profiles and universally low treatment-planning performance, with medical-domain models trailing general-purpose MLLMs substantially on multi-image tracks despite competitive diagnostic scores. These patterns are consistent with a capacity dilution effect: medical fine-tuning can narrow the diagnostic gap but may erode the compositional multi-image capability that rare-disease evidence integration demands.