Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning
This work addresses the need for interpretable AI in medical imaging to assist professionals, though it is incremental as it builds on existing VQA methods with new data and graph-based enhancements.
The authors tackled the problem of medical visual question answering (VQA) by collecting a large-scale dataset focused on chest X-ray images with detailed questions and proposing a method using multi-modal relationship graphs, achieving improved interpretability and performance.
Medical visual question answering (VQA) aims to answer clinically relevant questions regarding input medical images. This technique has the potential to improve the efficiency of medical professionals while relieving the burden on the public health system, particularly in resource-poor countries. Existing medical VQA methods tend to encode medical images and learn the correspondence between visual features and questions without exploiting the spatial, semantic, or medical knowledge behind them. This is partially because of the small size of the current medical VQA dataset, which often includes simple questions. Therefore, we first collected a comprehensive and large-scale medical VQA dataset, focusing on chest X-ray images. The questions involved detailed relationships, such as disease names, locations, levels, and types in our dataset. Based on this dataset, we also propose a novel baseline method by constructing three different relationship graphs: spatial relationship, semantic relationship, and implicit relationship graphs on the image regions, questions, and semantic labels. The answer and graph reasoning paths are learned for different questions.