PRIOR: Prototype Representation Joint Learning from Medical Images and Reports
This work addresses the challenge of fine-grained multimodal representation learning in medical imaging, offering incremental improvements for tasks like classification and retrieval.
The paper tackles the problem of learning joint representations from medical images and reports by proposing a prototype representation learning framework with global and local alignment, cross-modality reconstruction, and a non-auto-regressive generation paradigm, resulting in outperforming state-of-the-art methods on five downstream tasks across multiple datasets.
Contrastive learning based vision-language joint pre-training has emerged as a successful representation learning strategy. In this paper, we present a prototype representation learning framework incorporating both global and local alignment between medical images and reports. In contrast to standard global multi-modality alignment methods, we employ a local alignment module for fine-grained representation. Furthermore, a cross-modality conditional reconstruction module is designed to interchange information across modalities in the training phase by reconstructing masked images and reports. For reconstructing long reports, a sentence-wise prototype memory bank is constructed, enabling the network to focus on low-level localized visual and high-level clinical linguistic features. Additionally, a non-auto-regressive generation paradigm is proposed for reconstructing non-sequential reports. Experimental results on five downstream tasks, including supervised classification, zero-shot classification, image-to-text retrieval, semantic segmentation, and object detection, show the proposed method outperforms other state-of-the-art methods across multiple datasets and under different dataset size settings. The code is available at https://github.com/QtacierP/PRIOR.