Location-Aware Pretraining for Medical Difference Visual Question Answering
This work is significant for radiologists and medical AI developers by improving the accuracy of differential medical VQA, an incremental step towards more reliable diagnostic tools.
This paper addresses the challenge of medical difference VQA, where models must identify subtle visual variations between multiple medical images, similar to radiologists' comparative diagnostics. The authors introduce a pretraining framework with location-aware tasks (AREF, GCAP, CAREF) to enhance vision encoders, leading to state-of-the-art performance in detecting and reasoning about clinically relevant changes in chest X-ray images.
Unlike conventional single-image models, differential medical VQA frameworks process multiple images to identify differences, mirroring the comparative diagnostic workflow of radiologists. However, standard vision encoders trained on contrastive or classification objectives often fail to capture the subtle visual variations necessary for distinguishing disease progression from acquisition differences. To address this limitation, we introduce a pretraining framework that incorporates location-aware tasks, including automatic referring expressions (AREF), grounded captioning (GCAP), and conditional automatic referring expressions (CAREF). These specific tasks enable the vision encoder to learn fine-grained, spatially grounded visual representations that are often overlooked by traditional pre-training methods. We subsequently integrate this enhanced vision encoder with a language model to perform medical difference VQA. Experimental results demonstrate that our approach achieves state-of-the-art performance in detecting and reasoning about clinically relevant changes in chest X-ray images.