Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training
This addresses the burden on radiologists by improving automated medical imaging diagnosis through enhanced cross-modal learning, though it is incremental as it builds on existing Med-VLP methods.
The paper tackles the problem of insufficient cross-modal interaction in Medical Vision-Language Pre-training (Med-VLP) by proposing a unified framework with Multi-task Paired Masking and Alignment (MPMA), which integrates cross-modal alignment into joint image-text reconstruction, and it outperforms previous methods in all downstream tasks.
In recent years, the growing demand for medical imaging diagnosis has placed a significant burden on radiologists. As a solution, Medical Vision-Language Pre-training (Med-VLP) methods have been proposed to learn universal representations from medical images and reports, benefiting downstream tasks without requiring fine-grained annotations. However, existing methods have overlooked the importance of cross-modal alignment in joint image-text reconstruction, resulting in insufficient cross-modal interaction. To address this limitation, we propose a unified Med-VLP framework based on Multi-task Paired Masking with Alignment (MPMA) to integrate the cross-modal alignment task into the joint image-text reconstruction framework to achieve more comprehensive cross-modal interaction, while a Global and Local Alignment (GLA) module is designed to assist self-supervised paradigm in obtaining semantic representations with rich domain knowledge. Furthermore, we introduce a Memory-Augmented Cross-Modal Fusion (MA-CMF) module to fully integrate visual information to assist report reconstruction and fuse the multi-modal representations adequately. Experimental results demonstrate that the proposed unified approach outperforms previous methods in all downstream tasks, including uni-modal, cross-modal, and multi-modal tasks.