Multi-Task Learning for Visually Grounded Reasoning in Gastrointestinal VQA
This work addresses the need for more accurate and interpretable AI systems in medical diagnostics, specifically for gastrointestinal imaging, though it is incremental as it builds on existing models and datasets.
The researchers tackled the problem of improving visual question answering for gastrointestinal images by developing a multi-task framework that simultaneously handles answering questions, generating explanations, and linking text to visual regions, which substantially improved both answer accuracy and visual localization over single-task baselines.
We present a multi-task framework for the MediaEval Medico 2025 challenge, leveraging a LoRA-tuned Florence-2 model for simultaneous visual question answering (VQA), explanation generation, and visual grounding. The proposed system integrates three curated datasets: (1) Kvasir-VQA-x1 for question-answer learning, (2) a synthetically enriched explanation dataset offering structured medical reasoning, and (3) text-to-region pairs linking visual features with segmentation masks. This multi-task setup enables the model to jointly learn visual grounding, reasoning, and interpretation, producing responses that are both accurate and interpretable. Extensive evaluation demonstrates that our approach substantially improves over single-task baselines in both answer accuracy and visual localization, highlighting the effectiveness of grounded multi-task learning for medical VQA applications.