See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay
This addresses the challenge of grounding VLMs for better gameplay in interactive environments, but it is incremental as it builds on existing methods by testing symbolic representations.
The study tackled the problem of Vision-Language Models (VLMs) struggling to translate visual perception into precise actions in interactive environments by evaluating their performance with symbolic representations. Results showed that VLMs benefit from accurate symbolic information, but performance depends on model capability and scene complexity when symbols are self-extracted.
Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines. Our results indicate that all models benefit when the symbolic information is accurate. However, when VLMs extract symbols themselves, performance becomes dependent on model capability and scene complexity. We further investigate how accurately VLMs can extract symbolic information from visual inputs and how noise in these symbols affects decision-making and gameplay performance. Our findings reveal that symbolic grounding is beneficial in VLMs only when symbol extraction is reliable, and highlight perception quality as a central bottleneck for future VLM-based agents.