Semantically-Prompted Language Models Improve Visual Descriptions
This addresses the challenge of ambiguous visual descriptions in language-vision models, offering improvements for tasks like image classification and generation, though it appears incremental as it builds on existing methods.
The paper tackled the problem of generating specific and expressive visual descriptions by proposing V-GLOSS, which uses semantic prompting and a contrastive algorithm to improve descriptions and achieve strong zero-shot results on datasets like ImageNet and FGVC Aircraft.
Language-vision models like CLIP have made significant strides in vision tasks, such as zero-shot image classification (ZSIC). However, generating specific and expressive visual descriptions remains challenging; descriptions produced by current methods are often ambiguous and lacking in granularity. To tackle these issues, we propose V-GLOSS: Visual Glosses, a novel method built upon two key ideas. The first is Semantic Prompting, which conditions a language model on structured semantic knowledge. The second is a new contrastive algorithm that elicits fine-grained distinctions between similar concepts. With both ideas, we demonstrate that V-GLOSS improves visual descriptions and achieves strong results in the zero-shot setting on general and fine-grained image-classification datasets, including ImageNet, STL-10, FGVC Aircraft, and Flowers 102. Moreover, these descriptive capabilities contribute to enhancing image-generation performance. Finally, we introduce a quality-tested silver dataset with descriptions generated with V-GLOSS for all ImageNet classes.