Anatomical Structure-Guided Medical Vision-Language Pre-training
This work addresses interpretability and representation issues in medical AI for clinical applications, though it appears incremental as it builds on existing vision-language pre-training approaches.
The paper tackled the challenges of local alignment interpretability and insufficient representation learning in medical vision-language pre-training by proposing an Anatomical Structure-Guided framework, which outperformed state-of-the-art methods on five public benchmarks.
Learning medical visual representations through vision-language pre-training has reached remarkable progress. Despite the promising performance, it still faces challenges, i.e., local alignment lacks interpretability and clinical relevance, and the insufficient internal and external representation learning of image-report pairs. To address these issues, we propose an Anatomical Structure-Guided (ASG) framework. Specifically, we parse raw reports into triplets <anatomical region, finding, existence>, and fully utilize each element as supervision to enhance representation learning. For anatomical region, we design an automatic anatomical region-sentence alignment paradigm in collaboration with radiologists, considering them as the minimum semantic units to explore fine-grained local alignment. For finding and existence, we regard them as image tags, applying an image-tag recognition decoder to associate image features with their respective tags within each sample and constructing soft labels for contrastive learning to improve the semantic association of different image-report pairs. We evaluate the proposed ASG framework on two downstream tasks, including five public benchmarks. Experimental results demonstrate that our method outperforms the state-of-the-art methods.