Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images
This addresses the problem of limited medical AI applications due to fine-grained tasks and high domain knowledge demands, offering a novel method for chest radiology with significant performance gains.
The paper tackled the challenge of applying multi-modal foundation models to medical domains by proposing Knowledge-enhanced Auto Diagnosis (KAD), which leverages medical knowledge to guide vision-language pre-training on chest X-rays and reports, resulting in zero-shot performance comparable to fully-supervised models and superior to expert radiologists for three out of five pathologies, with few-shot fine-tuning outperforming existing approaches.
While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge. To address this challenge, we propose a novel approach called Knowledge-enhanced Auto Diagnosis (KAD) which leverages existing medical domain knowledge to guide vision-language pre-training using paired chest X-rays and radiology reports. We evaluate KAD on {four} external X-ray datasets and demonstrate that its zero-shot performance is not only comparable to that of fully-supervised models, but also superior to the average of three expert radiologists for three (out of five) pathologies with statistical significance. Moreover, when few-shot annotation is available, KAD outperforms all existing approaches in fine-tuning settings, demonstrating its potential for application in different clinical scenarios.