Saliency Guided Longitudinal Medical Visual Question Answering
This work addresses the challenge of interpreting clinically meaningful changes in paired medical images over time for applications in radiology and healthcare, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackled the problem of longitudinal medical visual question answering (Diff-VQA) by proposing a saliency-guided encoder-decoder model that uses keyword-conditioned saliency to enforce spatially consistent attention across time points, achieving competitive performance on metrics like BLEU, ROUGE-L, CIDEr, and METEOR without radiology-specific pretraining.
Longitudinal medical visual question answering (Diff-VQA) requires comparing paired studies from different time points and answering questions about clinically meaningful changes. In this setting, the difference signal and the consistency of visual focus across time are more informative than absolute single-image findings. We propose a saliency-guided encoder-decoder for chest X-ray Diff-VQA that turns post-hoc saliency into actionable supervision. The model first performs a lightweight near-identity affine pre-alignment to reduce nuisance motion between visits. It then executes a within-epoch two-step loop: step 1 extracts a medically relevant keyword from the answer and generates keyword-conditioned Grad-CAM on both images to obtain disease-focused saliency; step 2 applies the shared saliency mask to both time points and generates the final answer. This closes the language-vision loop so that the terms that matter also guide where the model looks, enforcing spatially consistent attention on corresponding anatomy. On Medical-Diff-VQA, the approach attains competitive performance on BLEU, ROUGE-L, CIDEr, and METEOR while providing intrinsic interpretability. Notably, the backbone and decoder are general-domain pretrained without radiology-specific pretraining, highlighting practicality and transferability. These results support saliency-conditioned generation with mild pre-alignment as a principled framework for longitudinal reasoning in medical VQA.