Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection
This work addresses zero-shot medical detection for clinical applications by improving alignment in vision-language models, representing an incremental advancement over existing methods.
The paper tackles the problem of coarse alignment between images and disease descriptions in zero-shot medical detection by proposing StructuralGLIP, which encodes prompts into a knowledge bank for fine-grained alignment, resulting in a +4.1% AP improvement over prior state-of-the-art methods across seven benchmarks and a +3.2% AP boost for fine-tuned models on endoscopy datasets.
Zero-shot medical detection can further improve detection performance without relying on annotated medical images even upon the fine-tuned model, showing great clinical value. Recent studies leverage grounded vision-language models (GLIP) to achieve this by using detailed disease descriptions as prompts for the target disease name during the inference phase. However, these methods typically treat prompts as equivalent context to the target name, making it difficult to assign specific disease knowledge based on visual information, leading to a coarse alignment between images and target descriptions. In this paper, we propose StructuralGLIP, which introduces an auxiliary branch to encode prompts into a latent knowledge bank layer-by-layer, enabling more context-aware and fine-grained alignment. Specifically, in each layer, we select highly similar features from both the image representation and the knowledge bank, forming structural representations that capture nuanced relationships between image patches and target descriptions. These features are then fused across modalities to further enhance detection performance. Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP improvement over prior state-of-the-art methods across seven zero-shot medical detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP on endoscopy image datasets.