Self-supervised vision-language pretraining for Medical visual question answering
This work addresses the challenge of limited training data in medical VQA, providing a solution for clinical applications, though it is incremental as it builds on existing pretrain-finetune paradigms.
The paper tackles medical visual question answering (VQA) by proposing a self-supervised pretraining method called M2I2, which integrates vision and language through masked modeling and contrastive learning, achieving state-of-the-art performance on three public medical VQA datasets.
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption dataset, and finetunes to downstream medical VQA tasks. The proposed method achieves state-of-the-art performance on all the three public medical VQA datasets. Our codes and models are available at https://github.com/pengfeiliHEU/M2I2.