LVLMs and Humans Ground Differently in Referential Communication
This work addresses a critical deficit in AI-human collaboration for effective partnership, though it is incremental in highlighting specific limitations rather than proposing a new solution.
The study investigated the ability of large vision-language models (LVLMs) to model common ground in referential communication tasks, revealing significant limitations in their interactive resolution of referring expressions compared to human-human interactions.
For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical. But this ability to collaborate remains limited by a critical deficit: an inability to model common ground. Here, we present a referential communication experiment with a factorial design involving director-matcher pairs (human-human, human-AI, AI-human, and AI-AI) that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels. We release the online pipeline for data collection, the tools and analyses for accuracy, efficiency, and lexical overlap, and a corpus of 356 dialogues (89 pairs over 4 rounds each) that unmasks LVLMs' limitations in interactively resolving referring expressions, a crucial skill that underlies human language use.