Measuring and Aligning Abstraction in Vision-Language Models with Medical Taxonomies
This addresses the need for safer and more clinically meaningful deployment of vision-language models in medical imaging, though it is incremental as it builds on existing VLMs with new evaluation and fine-tuning techniques.
The paper tackled the problem that standard flat metrics fail to distinguish between clinically minor and severe errors in vision-language models for chest X-ray classification, and introduced hierarchical metrics and mitigation methods that reduced severe abstraction errors to below 2% while maintaining competitive performance.
Vision-Language Models show strong zero-shot performance for chest X-ray classification, but standard flat metrics fail to distinguish between clinically minor and severe errors. This work investigates how to quantify and mitigate abstraction errors by leveraging medical taxonomies. We benchmark several state-of-the-art VLMs using hierarchical metrics and introduce Catastrophic Abstraction Errors to capture cross-branch mistakes. Our results reveal substantial misalignment of VLMs with clinical taxonomies despite high flat performance. To address this, we propose risk-constrained thresholding and taxonomy-aware fine-tuning with radial embeddings, which reduce severe abstraction errors to below 2 per cent while maintaining competitive performance. These findings highlight the importance of hierarchical evaluation and representation-level alignment for safer and more clinically meaningful deployment of VLMs.