Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information
This addresses the challenge of polysemous words in image-text matching for VWSD, offering an incremental improvement over existing methods.
The paper tackles the problem of visual word sense disambiguation (VWSD) by introducing an unsupervised approach that incorporates gloss information from lexical knowledge-bases, using Bayesian inference and context-aware definition generation with GPT-3. The results show significant performance increases, with the context-aware method achieving prominent improvement in out-of-dictionary examples.
Visual Word Sense Disambiguation (VWSD) is a task to find the image that most accurately depicts the correct sense of the target word for the given context. Previously, image-text matching models often suffered from recognizing polysemous words. This paper introduces an unsupervised VWSD approach that uses gloss information of an external lexical knowledge-base, especially the sense definitions. Specifically, we suggest employing Bayesian inference to incorporate the sense definitions when sense information of the answer is not provided. In addition, to ameliorate the out-of-dictionary (OOD) issue, we propose a context-aware definition generation with GPT-3. Experimental results show that the VWSD performance significantly increased with our Bayesian inference-based approach. In addition, our context-aware definition generation achieved prominent performance improvement in OOD examples exhibiting better performance than the existing definition generation method.