Probing Contextual Language Models for Common Ground with Visual Representations
This work addresses the problem of understanding visual grounding in language models for researchers in AI and NLP, but it is incremental as it builds on existing probing methods.
The paper investigates how well contextual language models align with visual representations by probing whether text-only representations can distinguish matching from non-matching image patches. The results show that language representations are effective for retrieving object categories and specific instances, with textual context playing a key role, but visually grounded models slightly outperform text-only ones and greatly underperform humans.
The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations. In this work, we consider a new question: to what extent contextual representations of concrete nouns are aligned with corresponding visual representations? We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Moreover, they are effective in retrieving specific instances of image patches; textual context plays an important role in this process. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans. We hope our analyses inspire future research in understanding and improving the visual capabilities of language models.