DALPHIN: Benchmarking Digital Pathology AI Copilots Against Pathologists on an Open Multicentric Dataset
This benchmark provides a standardized, publicly available evaluation platform for pathology AI copilots, enabling rigorous comparison against human pathologists across diverse diagnoses and expertise levels.
The paper introduces DALPHIN, the first multicentric open benchmark for pathology AI copilots, comprising 1236 images from 300 cases across 130 diagnoses, 6 countries, and 14 subspecialties. Human performance from 31 pathologists is compared against three AI copilots (GPT-5, Gemini 2.5 Pro, PathChat+), finding that PathChat+ achieved no statistically significant difference from expert-level performance in four of six tasks, while Gemini and GPT achieved this in two and one tasks, respectively.
Foundation models with visual question answering capabilities for digital pathology are emerging. Such unprecedented technology requires independent benchmarking to assess its potential in assisting pathologists in routine diagnostics. We created DALPHIN, the first multicentric open benchmark for pathology AI copilots, comprising 1236 images from 300 cases, spanning 130 rare to common diagnoses, 6 countries, and 14 subspecialties. The DALPHIN design and dataset are introduced alongside a human performance benchmark of 31 pathologists from 10 countries with varying expertise. We report results for two general-purpose (GPT-5, Gemini 2.5 Pro) and one pathology-specific copilot (PathChat+) for sequential and independent answer generation. We observed no statistically significant difference from expert-level performance in four of six tasks for PathChat, 2/6 tasks for Gemini, and 1/6 tasks for GPT. DALPHIN is publicly released with sequestered, indirectly accessible ground truth to foster robust and enduring benchmarking. Data, methods, and the evaluation platform are accessible through dalphin.grand-challenge.org.