Can Vision-Language Models Infer Speaker's Ignorance? The Role of Visual and Linguistic Cues
This addresses the problem of evaluating pragmatic competence in multimodal AI models for researchers in AI and linguistics, though it is incremental in scope.
This study tested whether vision-language models can infer speaker ignorance from contextual cues, finding that when provided with both visual and linguistic cues, Claude 3.5 sonnet showed more human-like inference by integrating them, while GPT-4o and Gemini 1.5 Pro favored literal interpretations.
This study investigates whether vision-language models (VLMs) can perform pragmatic inference, focusing on ignorance implicatures, utterances that imply the speaker's lack of precise knowledge. To test this, we systematically manipulated contextual cues: the visually depicted situation (visual cue) and QUD-based linguistic prompts (linguistic cue). When only visual cues were provided, three state-of-the-art VLMs (GPT-4o, Gemini 1.5 Pro, and Claude 3.5 sonnet) produced interpretations largely based on the lexical meaning of the modified numerals. When linguistic cues were added to enhance contextual informativeness, Claude exhibited more human-like inference by integrating both types of contextual cues. In contrast, GPT and Gemini favored precise, literal interpretations. Although the influence of contextual cues increased, they treated each contextual cue independently and aligned them with semantic features rather than engaging in context-driven reasoning. These findings suggest that although the models differ in how they handle contextual cues, Claude's ability to combine multiple cues may signal emerging pragmatic competence in multimodal models.