MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning
This work addresses the problem of improving clinical decision support in dermatology for healthcare providers, though it appears incremental as it builds on existing models like VGG16 and ViT-CLIP.
The paper tackled medical question answering in dermatology by proposing a weakly supervised learning approach with multimodal fusion, achieving multilingual capabilities and handling open-ended questions without predefined choices.
The MEDIQA-M3G 2024 challenge necessitates novel solutions for Multilingual & Multimodal Medical Answer Generation in dermatology (wai Yim et al., 2024a). This paper addresses the limitations of traditional methods by proposing a weakly supervised learning approach for open-ended medical question-answering (QA). Our system leverages readily available MEDIQA-M3G images via a VGG16-CNN-SVM model, enabling multilingual (English, Chinese, Spanish) learning of informative skin condition representations. Using pre-trained QA models, we further bridge the gap between visual and textual information through multimodal fusion. This approach tackles complex, open-ended questions even without predefined answer choices. We empower the generation of comprehensive answers by feeding the ViT-CLIP model with multiple responses alongside images. This work advances medical QA research, paving the way for clinical decision support systems and ultimately improving healthcare delivery.