Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations
This addresses lexical ambiguity in natural language understanding for LLMs, offering an incremental improvement in a domain-specific task.
The paper tackled visual word sense disambiguation by developing a framework using CLIP with dual-channel text prompting and image augmentations, improving MRR from 0.7227 to 0.7590 and Hit Rate from 0.5810 to 0.6220 on the SemEval-2023 dataset.
Ambiguity poses persistent challenges in natural language understanding for large language models (LLMs). To better understand how lexical ambiguity can be resolved through the visual domain, we develop an interpretable Visual Word Sense Disambiguation (VWSD) framework. The model leverages CLIP to project ambiguous language and candidate images into a shared multimodal space. We enrich textual embeddings using a dual-channel ensemble of semantic and photo-based prompts with WordNet synonyms, while image embeddings are refined through robust test-time augmentations. We then use cosine similarity to determine the image that best aligns with the ambiguous text. When evaluated on the SemEval-2023 VWSD dataset, enriching the embeddings raises the MRR from 0.7227 to 0.7590 and the Hit Rate from 0.5810 to 0.6220. Ablation studies reveal that dual-channel prompting provides strong, low-latency performance, whereas aggressive image augmentation yields only marginal gains. Additional experiments with WordNet definitions and multilingual prompt ensembles further suggest that noisy external signals tend to dilute semantic specificity, reinforcing the effectiveness of precise, CLIP-aligned prompts for visual word sense disambiguation.