VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders
This work addresses the challenge of costly fine-tuning for vision-language discriminative tasks, offering a zero-shot solution that could benefit researchers and practitioners in computer vision.
The authors tackled the problem of applying pre-trained text-to-image diffusion models to visual grounding without fine-tuning, and their VGDiffZero framework achieved strong performance on benchmarks like RefCOCO, RefCOCO+, and RefCOCOg.
Large-scale text-to-image diffusion models have shown impressive capabilities for generative tasks by leveraging strong vision-language alignment from pre-training. However, most vision-language discriminative tasks require extensive fine-tuning on carefully-labeled datasets to acquire such alignment, with great cost in time and computing resources. In this work, we explore directly applying a pre-trained generative diffusion model to the challenging discriminative task of visual grounding without any fine-tuning and additional training dataset. Specifically, we propose VGDiffZero, a simple yet effective zero-shot visual grounding framework based on text-to-image diffusion models. We also design a comprehensive region-scoring method considering both global and local contexts of each isolated proposal. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that VGDiffZero achieves strong performance on zero-shot visual grounding. Our code is available at https://github.com/xuyang-liu16/VGDiffZero.