Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?
This work addresses the problem of enhancing embodied knowledge in language models for AI applications, but it is incremental as it reveals limitations rather than proposing a new solution.
The study investigated whether visual grounding improves large language models' understanding of embodied knowledge, finding that vision-language models did not outperform text-only models on a benchmark with over 1,700 questions, and they performed worse in visual tasks compared to other sensory dimensions.
Despite significant progress in multimodal language models (LMs), it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. To address this question, we propose a novel embodied knowledge understanding benchmark based on the perceptual theory from psychology, encompassing visual, auditory, tactile, gustatory, olfactory external senses, and interoception. The benchmark assesses the models' perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. By comparing 30 state-of-the-art LMs, we surprisingly find that vision-language models (VLMs) do not outperform text-only models in either task. Moreover, the models perform significantly worse in the visual dimension compared to other sensory dimensions. Further analysis reveals that the vector representations are easily influenced by word form and frequency, and the models struggle to answer questions involving spatial perception and reasoning. Our findings underscore the need for more effective integration of embodied knowledge in LMs to enhance their understanding of the physical world.