Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answering
This work improves medical VQA for healthcare applications by enhancing cross-modal fusion, though it is incremental as it builds on existing transfer learning and attention mechanisms.
The paper tackles the problem of medical visual question answering by addressing the lack of labeled data and suboptimal feature compatibility in existing methods, achieving state-of-the-art performance with a multi-task pre-training approach and cross-modal self-attention module.
Due to the severe lack of labeled data, existing methods of medical visual question answering usually rely on transfer learning to obtain effective image feature representation and use cross-modal fusion of visual and linguistic features to achieve question-related answer prediction. These two phases are performed independently and without considering the compatibility and applicability of the pre-trained features for cross-modal fusion. Thus, we reformulate image feature pre-training as a multi-task learning paradigm and witness its extraordinary superiority, forcing it to take into account the applicability of features for the specific image comprehension task. Furthermore, we introduce a cross-modal self-attention~(CMSA) module to selectively capture the long-range contextual relevance for more effective fusion of visual and linguistic features. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods. Our code and models are available at https://github.com/haifangong/CMSA-MTPT-4-MedicalVQA.