CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training
This addresses the reliability gap in medical VQA for clinical decision-making, offering a practical, label-free solution for deployment, though it is incremental as it builds on existing vision-language models.
The paper tackled the problem of unreliable medical visual question answering under domain shift by introducing CoTBox-TTT, a test-time training method that adapts models at inference without retraining, resulting in a 12.3% accuracy increase on pathVQA when added to LLaVA.
Medical visual question answering could support clinical decision making, yet current systems often fail under domain shift and produce answers that are weakly grounded in image evidence. This reliability gap arises when models attend to spurious regions and when retraining or additional labels are impractical at deployment time. We address this setting with CoTBox-TTT, an evidence-first test-time training approach that adapts a vision-language model at inference while keeping all backbones frozen. The method updates only a small set of continuous soft prompts. It identifies question-relevant regions through a visual chain-of-thought signal and encourages answer consistency across the original image and a localized crop. The procedure is label free, and plug and play with diverse backbones. Experiments on medical VQA show that the approach is practical for real deployments. For instance, adding CoTBox-TTT to LLaVA increases closed-ended accuracy by 12.3% on pathVQA.