"Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Dataset
This work addresses the need for more detailed annotations to improve dialogue agents' ability to handle communication problems in instruction-following interactions, though it is incremental as it builds on prior datasets.
The authors tackled the problem of generating suitable instruction clarification requests (iCRs) in dialogue models by introducing CoDraw-iCR (v2), which extends existing identifiers with fine-grained, content-grounded annotations based on dialogue game items and actions, enabling better modeling and evaluation of repair capabilities.
Instruction Clarification Requests are a mechanism to solve communication problems, which is very functional in instruction-following interactions. Recent work has argued that the CoDraw dataset is a valuable source of naturally occurring iCRs. Beyond identifying when iCRs should be made, dialogue models should also be able to generate them with suitable form and content. In this work, we introduce CoDraw-iCR (v2), extending the existing iCR identifiers with fine-grained information grounded in the underlying dialogue game items and possible actions. Our annotation can serve to model and evaluate repair capabilities of dialogue agents.