Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare Diseases
For rare disease diagnosis where data is scarce, DDL provides a test-time adaptation method that significantly boosts localization accuracy and reliability without requiring retraining.
Dynamic Decision Learning (DDL) enables frozen large vision-language models to refine abnormality grounding decisions at test time, improving mAP@75 by up to 105% on rare-disease cases and outperforming supervised fine-tuning.
Clinical abnormality grounding for rare diseases is often hindered by data scarcity, making supervised fine-tuning impractical and single-pass inference highly unstable. We propose Dynamic Decision Learning (DDL), a framework that enables frozen large vision-language models (LVLMs) to refine their decisions across both language and visual spaces by optimizing instructions and consolidating predictions under visual perturbations. This process improves localization quality and produces a consensus-based reliability score that quantifies model confidence. Results on brain imaging benchmarks, including a rare-disease dataset with 281 pathology types across models ranging from 3B to 72B parameters, show that DDL improves mAP@75 by up to 105% on rare-disease cases and outperforms adaptation baselines and supervised fine-tuning. Furthermore, DDL demonstrates stronger calibration between reliability scores and localization accuracy under severe distribution shifts and increasing task difficulty. Code is available at: https://lijunrio.github.io/DDL/