Vision-language models for chest radiography do not always need the image
For clinicians and AI safety researchers, this work reveals that standard accuracy benchmarks are insufficient to validate image grounding, and proposes a necessary audit before clinical deployment.
The authors show that medical vision-language models often rely on text priors rather than images, with a text-only model achieving within 5.7 accuracy points of the best multimodal model and a 119B multimodal model being indistinguishable from a 7B text-only baseline. They introduce a causal audit to detect image grounding, finding that only 5 of 9 models use images selectively.
Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image. That inference is unsafe: a model exploiting finding-name priors scores like one that reads the scan, and no standard benchmark separates them. We introduce a causal audit that intervenes on the image, occluding the relevant region, occluding an irrelevant one, and swapping in another patient's same-label scan, and combines three behavioral metrics to test whether a correct answer depends on the image. Across nine systems, a text-only model with no image access reaches within 5.7 accuracy points of the best multimodal one, and a 119-billion-parameter multimodal model is statistically indistinguishable from a 7-billion text-only baseline. The audit splits the cohort into three models that ignore the image, one that is unstable, and five that use it selectively, for a subset of findings; the categories hold across a second dataset, resolution, and prompt phrasing. Against board-certified radiologists, a text-only model is statistically indistinguishable from a radiologist's accuracy while grounding at zero, whereas the image-using models ground at radiologist-comparable rates. Reported confidence flags ungrounded answers only when a model uses the image. Grounding audits, not accuracy, should gate clinical deployment.