Measuring How (Not Just Whether) VLMs Build Common Ground
This addresses the need for better evaluation of VLM reasoning in interactive communication, though it is incremental as it builds on existing grounding concepts.
The paper tackled the problem of evaluating vision-language models (VLMs) in interactive grounding contexts, rather than single-turn settings, by introducing a four-metric suite and testing it on 150 self-play sessions with three proprietary VLMs, finding that all models diverged from human patterns and that task success did not correlate with successful grounding.
Large vision language models (VLMs) increasingly claim reasoning skills, yet current benchmarks evaluate them in single-turn or question answering settings. However, grounding is an interactive process in which people gradually develop shared understanding through ongoing communication. We introduce a four-metric suite (grounding efficiency, content alignment, lexical adaptation, and human-likeness) to systematically evaluate VLM performance in interactive grounding contexts. We deploy the suite on 150 self-play sessions of interactive referential games between three proprietary VLMs and compare them with human dyads. All three models diverge from human patterns on at least three metrics, while GPT4o-mini is the closest overall. We find that (i) task success scores do not indicate successful grounding and (ii) high image-utterance alignment does not necessarily predict task success. Our metric suite and findings offer a framework for future research on VLM grounding.