IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning
This work addresses the problem of generating accurate captions without paired image-text data for researchers and practitioners in computer vision, representing a strong incremental advance in zero-shot captioning.
The paper tackles the modality gap in text-only training for zero-shot captioning by proposing IFCap, which uses image-like retrieval and frequency-based entity filtering to align text features with visual relevance, resulting in significant performance improvements over state-of-the-art methods in image and video captioning.
Recent advancements in image captioning have explored text-only training methods to overcome the limitations of paired image-text data. However, existing text-only training methods often overlook the modality gap between using text data during training and employing images during inference. To address this issue, we propose a novel approach called Image-like Retrieval, which aligns text features with visually relevant features to mitigate the modality gap. Our method further enhances the accuracy of generated captions by designing a Fusion Module that integrates retrieved captions with input features. Additionally, we introduce a Frequency-based Entity Filtering technique that significantly improves caption quality. We integrate these methods into a unified framework, which we refer to as IFCap ($\textbf{I}$mage-like Retrieval and $\textbf{F}$requency-based Entity Filtering for Zero-shot $\textbf{Cap}$tioning). Through extensive experimentation, our straightforward yet powerful approach has demonstrated its efficacy, outperforming the state-of-the-art methods by a significant margin in both image captioning and video captioning compared to zero-shot captioning based on text-only training.