Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests
This addresses the challenge of aligning model uncertainty with human-like interaction for language and vision models, though it is incremental as it builds on existing testbeds.
The paper tackled the problem of whether language models can recognize and express their own uncertainty through clarification requests, using reference games as a testbed. The results showed that models often struggle to translate internal uncertainty into adequate clarification behavior, even in simple tasks.
In human conversation, both interlocutors play an active role in maintaining mutual understanding. When addressees are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar addressee role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a good testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.