CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays
For researchers developing clinically reliable VLMs, CheXpercept exposes a critical gap in current domain adaptation by showing that models fail on deeper perceptual tasks.
CheXpercept is a benchmark for evaluating vision-language models on expert-level lesion perception in chest X-rays, spanning coarse detection, fine contour evaluation, and semantic attribute extraction. Benchmarking 14 models reveals adequate performance only at coarse levels, with medical VLMs showing no advantage over general-domain models.
The evaluation of vision-language models (VLMs) for chest X-ray (CXR) analysis has largely been limited to disease-presence classification without visual grounding. Such evaluations fail to verify the expert-level lesion perception necessary to ensure the clinical reliability of VLMs. To address these limitations, we introduce CheXpercept, a sequential, multi-level perception benchmark that mirrors a radiologist's cognitive workflow across coarse-level detection, fine-level contour evaluation and revision, and semantic-level attribute extraction. To ensure high clinical fidelity at scale, we construct the dataset using a semi-automated generation pipeline paired with a review by six medical experts. CheXpercept contains 10,400 QA items derived from 2,100 CXRs, covering seven clinically critical pulmonary and cardiac lesions. To demonstrate the current landscape of VLM perception, we benchmark 14 general and medical VLMs on CheXpercept. The models achieve adequate performance only at the coarse level, with accuracy degrading precipitously on deeper visual tasks. Notably, medical VLMs show almost no perceptual advantage over their general-domain counterparts, highlighting a systemic flaw in current domain adaptation. The code and dataset will be publicly available.